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SYSTEMS AND METHODS OP ERROR RESILIENCE IN A VIDEO DECODER 

Background Qf the Invention 

The invention it related to video decoding techniques. In particular, the mvemioo 
relates to systems and methods of error resilient decoding of a video bitstream. 

Pesc^jon^ftfteRgbtcd Art 

A variety of digital video compression techniques have arisen to transmit or to atoxe 
a video si cm] with a lower bandwidth or with less storage space. Such video compression 
techniques include international standards, such as HJ261, K263. H-263-r, H.263++, 
H-26T, MPEG-1, MPEG-2, MPEG-4, and MPEG-7. These compressioa techniques 
achieve relatively high compression ratios by discrete cosine transform (DCT) techniques 
and motion compensation (MC) techmques. among others. Such video compression 
techniques permit video bitstrcams to be efficiently earned across a variety of digital 
networks, such as wireless cellular telephony networks, computer networks, coble 
networks, via satellite; and the like. 

Unfortcnately for users, the various medioms used to carry or transmit digital video 
signals do not always work perfectly, and the transmitted data can bo corrupted or otherwise 
interrupted. Such corruption can include errors, dropouts, and delays. Corruption occurs 
with relative frequency in some transmission mediums; such as in wireless channels and in 
asynchronous transfer mode (ATM) ncrmtka. For example, data transmission in a wireless 
chancel can be corrupted by environmental noise, mnhjoath, sod shadowing. In another 
example, data transmission in an ATM network can be corrupted by network congestion 
and buffer overflow. 

Canuption in a data stream or bitstxcom that is carrying video can causa disruptions 
to the displayed video. Even the loss of one bit of data can result in a loss of 
synchronization with (he bhstrram, which results in the unavailability of subsequent bits 
undl a synshronizabOQ codeword is received. These errors in transmission can cause 
frames to be missed, blocks within a frame to be missed, and the Eke. One drawback to a 
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relatively highly compressed data stream is on increased susceptibility to corruption in the 
tmnsmiuion of the data stream carrying the video signal. 

Those in the en have sought to develop techniques to mitigate against the corruption 
of data to tbe bitstream. Far example, error concealment techniques can be used in an 
attempt to hide errors in missing or cocruptod blocks. However, conventional error 
concealment techniques can be relatively crude and unsophisticated. 

In another example, forward enox correction (FEQ techniques are used to recover 
corrupted bits, and thus reconstruct data in (he event of corruption. However, EEC 
techniques disadvantageousry introduce redundant data, which increases the bandwidth of 
the bitstream for the video or decreases the amount of elective bandwidth remaining for 
the video. Also, FEC techniques are computationally complex to implement In addition, 
coavenhonal FEC techniques are not compatible with the international standards, such as 
&261, K263. MPEG-2, and MPEG-*, bat instead, have to be implemented at u higher, 
"systems" level 

Summary of the Invenrion 
The invention is related to methods and apparatus that conccaJ errors m nnages of a 
corrupted video bitstream. One embodiment conceals error* in a missing or corrupted 
ratro-coded maexoblock by linearly imerpolating data from other nucroblocks that 
correspond to portions of the imago above and below the missing or corrupted macrobloclL 
One embodiment • can utilize substitute motion vectors for a cussing or corrupted 
predictive-coded macroblock. Another embodiment doubles the received motion vectors 
end references the doubled motion vectors to a previous-previous frame. Another 
embodiment adoptively selects which concealment or reconstruction technique is applied 
according to projected error estimates. Another embodiment conceals errors by replacing 
corrupted or missing data by combining concealment data in a weighted sum to reduce an 

One embodiment of the invention includes a video decoder that conc e a l s errors 
received in a video bitstream, tbe video decoder comprising on error detection circuit 
adapted Co detect errors in (he video bitstream; a memory device configured to provide an 
indication of an- error in a portion of a video bitstream corresponding to a portion in on 
image; a control circuit configured to be responsive to an indication of (he error in a first 
portion of tbe imago, where the control circuit is further configured to detect if a second 
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portion above the first portion in the imago and if a third portion below the Hist portion in 
the image are error-free, where the control circuit is further configured to interpolate 
between conesponding data in the second portion of the imago and corresponding date in 
the third portion of the data to conceal chc error. 

Another embodiment according to the invention intrudes a video decoder that 
adap lively conceals erron received in a video bitstream, die video decoder comprising: o 
memory module adapted to maintain error value a for selected portions of an image; a 
plurality of error resilience modules that generate images In response to errors; a prediction 
mo lule adapted to generate a plurality of predictions of error values corresponding to the 
plurality of error resilience modules; a control module adapted receive an indication of an 
error in the video bitstream and, in response, to select an error resilience modulo from the 
error resilience module based on a CO nip arisen of the predictions of error values. 

One embodiment of (be invention includes a video decoder that conceals errors 
received in a video bitstream, the video decoder comprising: a memory module adapted to 
Tfraintjitii error variances for selected portions of an image; a plurality of error resilience 
modules that generate images in response to errors; a prediction module adapted to generate 
a plurality of weights corresponding to the plurality of error resilience modules; n control 
module adapted receive an indication of an error in the video bitstream and, in response, to 
combine outputs of selected error resilience modules with the weights from the prediction 
module to conceal the error. 

One embodiment of tba invention includes an optimizer circuit that selectively 
applies on error concealment technique from among a plurality of error concealment 
techniques comprising: means for maintaining an estimated error relating to at least a 
portion of an image; means for using the estimated error to generate a plurality of projected 
error estimates corresponding to application of an error concealment technique; and means 
for selecting the error concealment technique that provides the lowest projected error 
estimate. 

One embodiment of the invention includes a method of concealing errors in a video 
decoder comprising: detecting an error in a first portion of o video bitstream that is intra* 
coded; determining that a second portion of an image above the first portion, and a third 
portion of the imago below the first portion arc not corrupted; and interpolating pixels in 
the first portion between a first horizontal row of pixels in the second portion and a second 
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horizontal row of pixels in the third portion Lo conceal errors when the second portion and 
the third portion are sot corrupted. 

One embodiment of the invention includes a method of concealing mors in a video 
decoder comprising: detecting an error m a firel portion of a video bitstream that is 
prodicn've-coded; providing a substitute motion vector when the error relates to a standard 
motion vector; using a first reference portion of a previous frame with the substitute motion 
vector to reconstruct when the first reference portion is available; and using a second 
reference portion of a second frame that is prior to the previous frame when the first 
reference ponton oftho previous frame is not available. 

One embodiment of the invention includes a method of adapt i very producing a 
video image comprising: receiving video data for a frame; determining whether the video 
data is mtnvxoded or predictive-coded; when the video data is intra -coded: detennirring 
whether the mtra-coded video data corresponds to an error, concealing the error when the 
inrra-ceded video data corresponds to the error; setting an error value that is associated with 
at least a portion of the video packet to a first predetermined value when the mtra-coded 
video data corresponds lo rhe error; resetting the error value when no error for the mtra- 
coded video data is detected; and using the intrn-coded video dam when no error for the 
mtra-coded video dam is delected; when the video data is predictive-coded, taermining 
whether the predictive-coded video data corresponds m an error; when the predictive-coded 
video data corresponds to an error: using the prcdictivc-codcd video data when no error for 
the predictive-coded video data is detected and the associated error value is reset; projecting 
8 first estimated error corresponding to use of die predictive-coded video data when no 
error is detected for the predictivo-coded video data and the associated error value is not 
reset; projecting a second estimated error corresponding to use of a first predictive-coded 
error concealment technique when no error is detected for the predictive-coded video data 
and the ossocialed error vahie is not reset; selecting between the use of the predictive-coded 
video data and the use oftho Qrat predictive-coded error coaceaJracnt tehmauc based on a 
comparison between the first projected estimated error and the second projected estimated 
error, and updating the error value according to which of the pedictive-ccdcd video data 
and the first predictive-coded error concealment technique is selected; and when the 
predictive-coded video data corresponds to an error: applying a second predictive-coded 
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error concsahnent tec hniq ue; and updaiing the error value according to the second 
predictive-coded error concea lmen t technique. 

One embodiment of the invention includes e method of producing a video image 
comprising: receiving data for a video frame; determining whether the video frame is a 
predictive-cod cd frame or is an mtra-coded frame; perforrrring the folio wing when (he video 
frame is the predictive-coded frame: detttnnuung whether a group of video data from the 
video frame corresponds to an error; when (here is no error in the group of video data: 
determining whether the group of video data is bun-coded or predktivo-coded; intra- 
decoding the group of video data when the group of video data is intra coded; resetting an 
error variance associated with al least a portion of the group of video data when the group 
of video data is intra coded; using a find weighted cum to reconstruct a portion of an image 
cc CT cspondmg to the group of video dfito when the video data is intra coded, where the first 
weighted sum combines results of at least a first and a second technique; and updating the 
error variance according to the first weighted sum used to reconstruct the portion of the 
image; and when (here is an error in the group of video data: concealing the error in the 
portion of the image corresponding to the group of video data; and irpdairng the error 
variance according to the error concealment. 

One embodiment of the invention includes a method of selecting an error 
concealment technique from among a plurality of error concealment techniques comprising: 

error to generate a plurality of projected error esti m ates corresponding to application of an 
error concealment technique; and selecting the error conceahnent technique that provides 
tibe lowest projected error estimate. 

Tbo Invention is related to methods and apparatus that recover usable video data 
from partrany corrupted data. Embodiments inspect corrupted data packets and identify (he 
location or locations of an error, whether the corrupted data packet contains data expected 
to be error-free, and whether the error-free data should be used. Decoding of a packet in 
both the forward direction and the backward direction can bo used to locaie a position of an 
error. Imra-ooded macroblociu can also be recovered. A decoder can elect to use or to 
drop an mtra-coded macroblock recovered from a corrupted data packet according to 
further criteria that is applied to the nxoveced rnrra-coded macroblock. One embodiment 
inspects video b i tstrca m data that has been encoded with an optional date p a rt i tio nin g 
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feature enabled, and retrieves specified data in areas of a corrupted packet that are expected 
to he free from error. 

One embodiment of the invention includes a circuit adapted to recover useful dam 
from a video packet that in at least partially corrupted, the circuit comprising: a decoding 
circuit configured to decode 0 video packet in a forward direction and in a backward 
direction, where the decoding circuit detects bit locations of errors first encountered in the 
forward direction and in the backward direction; a counter adapted to maintain a count of 
complete macrobfocka decoded in the forward direction and in the backward direction; and 
a control circuit adapted to discard at least a portion of the video packet that corresponds to 
an overlapping region, where the control circuit is further configured to farther discard 
additional data corresponding to u backtracking amount when there is no ova lapping 
region, where (he control circuit is further configured to discard information in incomplete 
macroblocks, and where the control circuit is adapted to permit use of 01 least a portion of 
the rerairining data 

One embodiment of (he invention includes a circuit adapted to recover useful data 
from a video packet that ts at least partially corrupted, the circuit comprising: a data parsing 
circuit adapted to determine whether a video packet is encoded with data partitioning 
enabled; on error checking circuit configured to determine whether an error exists ahead of 
a motion marker of the video packet; and a decoder coupled to (he data parsing circuit and 
to the error checking circuit, where the decoder is adapted to decode at least a rxwuonof the 
data in the corrupted video packet ahead of the motion marker when data parsing circuit 
indicates that the video packet is encoded with data partm'oning enabled and when the error 
checking circuit indicates mat the error docs not exist ahead of the motion marker. 

One embodiment of the invention includes a circuit adapted to recover useful data 
from a video packet that is at least partially corrupted, the circuit comprising: means for 
receiving the video pocket; means for ending without recovering data when corruption is 
detected in at least one of a video packet header of the video packet, a DC portion of the 
video packet, and a motion vector portion of the video packet; means for rnMahng 
decoding of the video packet in a forward direction; means for maintaining a first count of a 
number of macroblocks decoded without error in the forward direction; means for storing 
codewords decoded in the forward direction; means for storing a first hit location when an 
error is first detected in the forward direction; means for mtriating decoding of the video 
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packet in a reverse direction; means for maintaining a second count of a cumber of 
macroblocks decoded without error in I be reverse direction; means Car etc ring codewords 
decoded in the reverse direction; means for storing a second bit location when an error is 
first detected in the reverse direction; means for determmhi£ if there is an overlapping 
region, where the overlapping region corresponds to a region identified in both the forward 
direction and in the reverse direction as having an error; means for discarding the data in 
the overlapping region and for using tho data in a remaining portion of (he video packet if 
there is an overlapping region; and means for discarding the data between a first 
backtracking amount ahead of the first error location in the forward direction and a second 
b acktr acking amount behind the second error location in the first location, and fox 
recovering the remaining portion of the video packet if there is no overlapping region. 

One embodiment of the invention includes a method of recovering useful data from 
a video packet that has been corrupted, the method comprising: receiving the video packet; 
ending without recovering data when corruption is detected in a video packet header of the 
video packet; ending without recovering dam when corruption is detected in a DC portion 
of the video packet; ending without recovering data when corruption is detected in a motion 
vectOT portion of the video packet; initiating decoding of the video packet in a forward 
direction; maintaining a first count of a number of macron locks decoded without error in 
the forward direction; storing codewords decoded in the forward direction; storing a first bit 
location when an error is Ontt detected in the forward direction; initiating decoding of the 
video packet in a reverse direction; maintaining a second count of a number of macroblocka 
decoded without error in the reverse direction; storing codewords decoded in the reverse 
direction; storing a second bit location when an error is first detected In the reverse 
direction; determining if there is an overlapping region, where the overlapping region 
corresponds to a region identified in both the forward direction and in the reverse direction 
as having an error; if there is an overlapping region, discarding the data in the overlapping 
region and using the data in a remaining portion of the video packet; and if there is no 
overlapping region, discarding the data between a first backtracking amount ahead of the 
first error location in the forward direction and e second backtracking amount behind the 
second error location in tile first location, and recovering the remaining portion of the video 
packet 
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One embodiment of the invention include* method for recovering data in a 
corrupted video packet comprising: inspecting the video packet to de ter mine whether the 
video packet was encoded with data partitioning enabled; dcJenruning whether an error 
oasts ahead of a motion marker of the video pocket; and decoding at least a portion of the 
data in the corrupted video packet ahead of the motion marker when the video packet was 
encoded with data partitioning enabled and when the error does not exist ahead of the 
motion marker. 

The invention is feinted to methods and apparatus thai advantageously reconstruct 
and decode video data, such as video object planes (VOPs), using forward error correction 
(FEC) codes embedded in the video cdnrurara Advantageously, the original video data can. 
be recovered oven when portions of the video bitstream are corrupted or lost during 
transmission. Further advantageously, the methods and apparams disclosed arc backward 
compatible with video bitstreams thai are compliant with standard syntax, thereby allowing 
a decoder to achieve compatibility with both standard video bitstreams and video bitstreams 
embedded with FEC codes. la one embodiment, a decoder retrieves the FEC codes from a 
user data video packet. To save bandwidth, on encoder can provide FEC codes 
corresponding to o subset of the video data, and the decoder eon receive end interpret 
mfheartons as to which data the provided FEC codes correspond. 

One embodiment of the invention includes a video decoder adapted to reconstruct 
corrupted video data co m p ri sing? a receiver circuit adapted to receive a video bitstream; a 
buffer coop led to the receiver circuit, where tho buffer is adapted to store at least a portion 
of the video bitstream; a parsing circuit adapted to distinguish video data from forward 
error correction (FEC) coder, an error inomumng circuit configured to delect corruption in 
the video data; and an FEC decoder adapted to receive the video data and the FEC codes, 
where (he FEC decoder is configured to remove tho corruption in the video data to which 
the FEC codes apply. 

One embodiment of (he invention includes a video decoder that decodes a video 
bitstream that includes forward error correction (FEC) codes, the video decoder 
compe lli ng ! means for receiving the video bititream, which includes both video data and 
FEC codes; means for retrieving video data from the video bitstream; means for 
detenrnning if there is corruption in a portion of (he video data retrieved; means for 
retrieving FEC codes from the video bitstream in response to a detection of corruption, and 
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means for using the FEC codes to reconstruct the portico of the video dula such that the 
portion of the video data is recovered without corruption. 

One embodiment of Ore invention includes a process of decoding a video bitstream 
that includes forward error correction (FEC) codes, (he process comprising: receiving the 
video bitstream. which includes both video data and FEC codes; retrieving video data from 
the video bitstream; determining if there is corruption in a portion of the video data 
retrieved; retrieving FEC codes from the video bitstream in response to a detection of 
corruption; end using the FEC codes to reconstruct the portion of the video data such (hat 
the portion of the video dam is recovered without corruption. 

One embodiment of (he invention memoes a process of decoding a video bitstream 
thai be hides forward error correction (FEC) codes, the process comprising: receiving the 
video bitstream, which includes both video data and FEC codes; retrieving video data from 
the video bitstream; determining if FEC codes mat correspond to the retrieved video data 
am available; retrieving FEC codes from the video bitstream when the FEC codes are 
available; and using the FEC codes to decode the portion of the video data such, thai the 
portion of the video data is r e c overed without corruption. 

The invention is related to methods and apparatus that manage data received in a 
video bitstream. Video data received from the video bitstream is placed in a ring buffer, 
which is also known as a circular buffer. Data logging raformalion. is also stored in the ring 
buffer, hi one embodiment, die data Jogging infonoation is stored synchronously or in 
alignment with the corresponding video dala> The data logging information can include a 
status of whether the conespemding video data with or without error. The ring buffer 
provides fast and efficient access to video bitstream data that may be accessed multiple 
tiroes in multiple directions by modules that extract data from the video bitstream in an 
cu&r resilient znsnocr* 

One embodiment of the invention includes a data buffer circuit for a video decoder 
comprising: a receiver circuit adapted to receive a video bitstream; a ring buffer adapted to 
store the video bitstream; and an error resilience module adapted to retrieve data from (he 
ring buffer. 

One embodiment of me invention includes a data buffer circuit for a video decoder 
comprising: means for receiving a video bitstream; means for uupecting the video 
bitstream for error; means for storing the video bitstream in a ring buffer regardless of an 
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error indication; means for strong ^ fogswg infonnalkm conrspocding to video 
bit strain data in (be ring buffer in nn aligned manna with tfcc conesponding video 
bintream data; and means for automaltcally retrieving both a portion of the video bitstream 
and Q corresponding portion of the data loggias information from tbo ring buffer in 
response to a request for data. 

One embodiment of the invention includes a method of accessing information from 
a video bitstream comprising receiving a video bitstream; inspecting (be video bitstream 
for error, storing the video bitstream in a ring buffer regardless of an error indication; 
storing data logging information corresponding to video bhstream data in the ring buffer in 
an aligned manner with (he corresponding video bitstream data; and automatically 
retrieving both a portion of (he video brtstream sod a corresrKmding portion of the data 
logging in formation from the ring buffer in response to a request for data 

Pricf Pcscrtptjon vf to Pnmrmn 

These and other features of the invention will now be described with reference to 
the drawings summarized below. These drawings and the associated description are 
provided to illustrate preferred embodiments of the invention and are not intended to limit 
the scope of the invention. 

Figure 1 illustrates a networked system for implementing a video distribution 
system in accordance with one embodiment of the invention. 
. Figure 2 ilrusrrsrcs a sequence of frames. 

Figure 3 is a flowchart generally Ulurrratfog a process of concealing errors or 
missing data in a video bitsUeam. 

Figure 4 illustrates a process of temporal concealment of missing morion vectors. 

Figure 5 is a flowchart generally illustrating a process of adaptrvely concealing 
errors in a video bitstream. 

Figure 6 is a flowchart generally illustrating a process that can use weighted 
predictions to compensate for errors in a video bitstream. 

Figure 7A Uru strata a sample of a video packet with DC and AC components for an 

I-VO?. 

Figure 7B illustrates a video packet for a P-VOP. 

Figure 8 illustrates an example of discarding a co rrupt ed mac rob lock. 
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Figure 9 Is a flowchart that generally illustrates a process according to an 
embodiment of (he invention of partial RVLC decoding of discrete cosine transform (DCT) 
portions of corrupted packets 

Figures 10-13 Ulustnae partial RVLC decoding strategies. 

Figure 14 illustrates o partially corrupted video packet with at least one infra-coded 
macrobiock. 

Figure IS illustrates a sequence of raataoblocks with AC prediction. 

Figure 16 illustrates a bit structure for an MPEG-4 data partitioraog packet 

Figure 17 illustrates one example of a tradeoff between block error rate (BER) 
correction capability versus overhead. 

Figure 18 illustrates a video birstrcam with systematic FEC data. 

Figure 19 is a flowchart generally illustrating a process of decoding systematically 
encoded FEC data m a video bitstream. 

Figure 20 is a block diagram generally illustrating one process of using a ring buffer 
is error resilient decoding of video data. 

Detailed Description of Preferred Embodiments 

Although this invention will bo described in- terms of certain preferred 
embodiments, other embodiments that are apparent to those of ordinary skill in the art, 
including embodiments that do not provide all of Ibe benefits and features set forth herein, 
arc also within the scope of this invention. Accordingly, the scope of (ho invention is 
defined only by reference to the appended claims. 

The display of video can consume a relatively large amount of band width, 
especially when the video is displayed in real time. Moreover, when the video Wtstrezm is 
wirelessly transmitted or is transmitted over a congested network, packets may be lost or 
unacccptaWy delayed. Even when a packet of data in a video bitstream is received, if the 
packet is not timely received due to network congestion and the like, the packet may not be 
usable for doxxhng of the video bitstream in real rime. Embodiments of the invention 
advantageously compensate for and conceal errors that occur when packets of data in a 
video bitstream arc delayed, dropped* or lost. Some embodiments reconstruct the original 
data from other data. Other embodiments conceal or hide the result of errors so that a 
corresponding display of the video bitstream exhibits relatively fewer errors, thereby 
effectively increasing the sigoal-to-noise ratio (SNR) of the system. Further 
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Advantageously, embodiments of the invention can remain downward compatible with 
video bUsttearos that are compliant with existing video encoding standards. 

Figure I flhislrates a networked system far implementing a video distribution 
system in accordance with one embodiment of the invention. An encoding computer 102 
receives a video signal, which is to be encoded to a relatively compact and robust format. 
The encoding computer 102 can correspond to a variety of ma c hi ne types, including general 
purpose computers that execute software and to specialized hardware. The encoding 
computer 102 con receive a video sequence from a wide variety of sources, such as via o 
satellite receiver 104, a video camera 106, and a video conferencing terminal 108. The 
video camera 106 can correspond to a variety of camera types, such as video camera 
recorders, Web cams, cameras built into wireless devices, and the like, Video sequences 
can also be stored in a data store 110. The data store 110 can be interna] to or external to 
the encoding computer 102. The data store 110 can include devices such as tapes, bard 
disks, optical disks, and the like. It win be understood by one of ordinary skill in the art 
that a data store, such as the data store 110 illustrated in Figure 1, can store uoencoded 
video, encoded video, or both, hi one embodiment, the encoding computer 102 retrieves 
un encoded video from a dam store, such aa the data stem 110, encodes the an encoded 
video, and stores the encoded video to a data store, which can be the came data store or 
another data store. It will be understood mat a source for the video can include a source 

The encoding computer 102 distributes the encoded video to a receiving device, 
which decodes the encoded video. The receiving device can correspond to a wide variety 
of devices mat can display video. For example, the receiving devices shown in the 
illustrated networked system include a cell phone 1 12, a personal digital assistant (PDA) 
114, a laptop computer 116, and a desktop computer US. The receiving devices can 
communicate with the encoding computer 102 through a communication network 120, 
which can correspond to a variety of communication networks including a wireless 
conimumcanoa network. It will be understood by one of ordinary skill in the art that a 
receiving device, such as the cell phone 1 12, can also be used to transmit a video signal to 
(he encoding computer 102. 

The encoding computer 102, as well as a receiving device or decoder, can 
correspond to a wide variety of computers. Per example, the encoding computer 102 can 
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be any microprocessor or processor (hereinafter referred to as processor) controlled device, 
including, but not limited to o terminal device, such as a personal computer, a workstation, 
a server, a client, a mini computer, a main-frame computer, a laptop cornpater, a network of 
individual computers, a mobile computer, a pofco top computer, a band odd computer, a set 
top box for a TV. on interactive television, an interactive kiosk, a personal digital as six lam 
(PDA), an interactive wireless communications device, a mobile browser, a Web enabled 
cell phone, or a combination thereof. The eornputerrnay father possess input devices such 
as a keyboard, a mouse, a trackball, a touch pod, or a touch screen and output devices such 
as a computer screen, printer, speaker, or other input devices now in existence or later 
developed. 

The encoding computer 102, as wdl as a decoder, described can correspond to a 
uniprocessor or multiprocessor machine. Additionally, the computers can include an 
addressable storage medium or computer accessible medium, such as random access 
memory (RAM), an electronically erasable pro g ramm able read-only memory (EEPROM), 
hard disks, floppy disks, laser disk players, digital video devices, Compact Disc ROMs, 
DVD-ROMs, video tapes, audio tapes, magnetic recording tracks, electronic networks, and 
other techniques to transmit or store electronic content such as, by way of example, 
programs and d a t a. In one embodiment, the computers are equipped with a network 
cornmumcatioa device such as a network interface card, a modem, InthvRed (IR) port, or 
other network c rr o npfi i fffi device suitable for connecting to a nctwoOc. Ftirthennnnj, the 
computers execute an appropriate operating system, such as Linux, Unix, Microsoft® 
Windows® 3.1, Microsoft® Windows® 95, Microsoft® Windows® 98, Microsoft® 
Windows® NT. Microsoft® Windows® 2000. Microsoft® Windows® Me, Microsoft® 
Windows® X?, Apple® MacOS®, IBM® OS/2®, Microsoft® Windows® CE, or Palm 
OS®. As is conventional, the appropriate operating system may advantageously include a 
communications protocol implementation, which handles all incoming and outgoing 
message traffic passed over the network, which can include » wireless network. In other 
ern J b od rm eals, while the operating system may differ depending on the type of computer, 
the operating system may continue to provide the appropriate co ram uru cations protocols 
n e c es sa r y to establish communication links with the network. 

Figure 2 ilrustrates a sequence of frames. A video sequence includes multiple video 
frames taken at interval*. The rate at which the frames are displayed is referred to as the 
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frame rate. In addition to techniques nsed to camp rets still video, motion video techniques 
relate a frame at I hue k to a fame at bine Jt-l lo further compress the video information into 
relatively small amounts of data. Dowever, if the frarao at time *-l is not availabjo dno to 
on error, such as a transmission error, conventional video techniques may not be able to 
property decode the frame at time k. As will be explained later, embodiments of the 
invention advantageously decode the video stream in a robust m inner such that the frame at 
lime* can be decoded even when the frame at time h\ is not available. 

The frames io a sequence of frames can correspond to either interlaced frames or to 
non-interlaced frames, i.e, progressive frames, m an interlaced frame, each frame is made 
of two separate fields, which arc interlaced together to create the frame. No such 
interlacing is performed in a non-interlaced or progressive frame. Wfcilo illustrated in the 
context of non-interlaced or progressive video, the skilled artisan will appr ec i ate that the 
principles and advantages described herein are applicable to both interlaced video and non- 
interlaced video, m addition, while certain embodiments of the invention may be described 
only in the context of MPEG-2 or only in the context of MPEG-4, (he principles and 
advantages described herein are applicable to a broad variety of video standards, including 
EL261, &J263, MPEG-2, and MPEG-4, as well as video standards yet to be developed. In 
addition, while certain embodiments of the invention may describe error concealment 
techniques in the context of, for example, a maaoblock, the skilled practitioner will 
appreciate that the techniques described herein can apply to blocks, macrobloclcs, video 
object planes, lines, individual pixels, groups of pixels, and the like. 

The standard is defined in "Coding of Audio-Visual Objects: Systems," 

14496-1, ISO/IEC JTCMSC29/WG31 N2501, November 1998, and "Coding of Audio- 
Visual Objects: Visual," 14496-2, ISO/IEC JTC1/SC29/WG11 N2502, November 1998, 
and the MPEG-4 Video Verification Model is defined in ISO/TEC JTC l/SC 29/WG H, 
-MPEG-4 Video Verification Model 17.0," ISO/IEC JTCI/SC29AVG11 N3515, Beijing, 
China, July 2000. 

In an MPEG-2 system, a frame is encoded into multiple blocks, and each block is 
encoded into six macrob locks. The mactoblocks mctude information, such as luminance 
and color, for composing a frame. In addition, while a frame may be encoded as a still 
frame, La, an Lota-coded Game, frames in a sequence of frames can be temporally related 



-14- 



(67) 



JP 2004-528752 A 2004.9.16 



WO01T0717J* PCTAK02.-OAW5 

to each other, i.e., predictive-coded frames, and the macroblocks can relate o section of one 
frame at one time to R section of another frame at another time. 

ha an MPEG-4 system, c frame in a sequence of frames is further encoded into a 
number of video objects known as video object planes (VOPs). A frame cm be encoded 
into a single VQF or (n multiple VOPs. h one system, such as a wireless system, each 
frame includes only one VOP so thai a VOP is a frame. The VOPs are transmitted to a 
receiver, where they are decoded by a decoder back into video objects for display. A VOP 
can correspond lo an intm-coded VOP (l-VOP), to a pmlicirvr-coded VOP (P-VOP) to a 
bidiitctic^ry-pjed'tctivo coded VOP (B-VOP), or to a eprito VOP (S-VOPX An I- VOP is 
not dependent on information from another frame or picture, i.e., an I-VOP is 
independently decoded. When a frame consists entirely of 1-VOPs, the frame is called an I- 
Frame, Such frames arc commonly used in xirooliorti such as a scene change. Although the 
lack of dependence on content from another frame allows an I-VOP to be robustly 
transmitted and received, an I-VOP dlsadvantageously consumes a relatively large amount 
of data or data bandwidth as compared to a P-VOP or B-VOP. To efficiently compress and 
transmit video, many VOPs in video frames correspond to F- VOPs. 

A P-VOP efficiently encodes a video object by referencing the video object to a pact 
VOP. Lc, to a video object (encoded by a VOP) earlier in lime. Ibis past VOP is referred 
to as a reference VOP. For example, where an object in a frame ai tune k is related to an 
object in a frame at time motion compsmatkra encoded in & P-VOP can be used to 
encode the video object with less mfonaaticn than with an I-VOP. The reference VOP can 
be either an 1-VOP or a P-VOP. 

A B-VOP uses both a pest VOP and a future VOP as reference VOPs. In a real-time 
video bitstream, a B-VOP should not be used. However, the principles and advantages 
described herein can also apply to a video bitstream with B-VOPs. An S-VOP is used to 
display anhuatod objects. 

The encoded VQPs are organized into macroblocks. A macro block includes 
sections far storing luminance (brightness) components and sections for storing 
chrominance (color) components. The macroblocks me transmitted and received via the 
communication network 120. It will be understood by one of ordinary skill in the art that 
the communication of the data can farther include other communication layers, such as 
modulation to and demodulation from code division multiple access (CDMA). It will be 
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understood by one of ordinary skill in the art thai the video bilstream can also include 
corresponding audio information, which is also encoded and decoded. 

Figure 3 is a flowchart 300 generally illustrating a process of conceabqg errors or 
missing data in a video bitstrctm. The errors can correspond to o variety of problems or 
unavailability including a loss of data, a corruption of data, a header error, a syntax error, a 
delay In receiving data, and the like. Advantageously, the process or Figure 3 is relatively 
unsophisticated to implement and can be execnted by relatively slow decoders. 

Upon the detection of an error, the process starts at a first decision block 304. The 
first decision block 304 determines whether the error retires to intra -coding or predictive- 
coding. It will be understood by the skilled practitioner that the krtra-ccding or predictive- 
coding can refer to frames, to macrobiotics, to video object planes (VOPs), and the like. 
While illustrated in the contact of inaeroblocks, the skilled artisan will appreciate that tho 
principles and advantages described in Figure 3 also apply to video object planes and the 
like. The process proceeds from the first decision block 304 to a first state 308 when the 
error relates to an intra-coded maaobkxt When the error relates to a predictive-coded 
mooroblock, the process proceeds from the first decision block 304 to a second decision 
block 312. Il will be understood thai the error for a predictive-coded macroblock can arise 
from a missing macroblock in a present frame at time /, or from an error in a reference 
frame al time M from which motion is referenced. 

In the first state 308, the process interpolate! or spatially conceals the error in the 
rntra-coded macroblock, termed a missing macroblock. In one embodiment, the process 
conceals the error in the missing macroblock by linearly mterpolatmg data from an upper 
macroblock thai is intended to be displayed "above" (he missing macroblock in the image, 
and from a lower macroblock that is intended to be displayed tseiow" the missing 
macroblock in the image. Techniques other than linear interpolation can also be used. 

For example, the process can vertically linearly interpolate using a tine denoted lb 
copied from the upper macroblock and a line denoted It copied from the lower macroblock. 
In one embodiment, tho process uses the lowermost hne of the upper macroblock aj lb and 
the topmost hue of the lower macroblock as h. 

Depending on the circumstances, the upper macroblock and/or the lower 
macroblock may also not be available. For example, the upper macroblock and/or (he 
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lower macroblock may have an cnor. Ia addition, *■ missing macroblocl: may bo located 
at the upper boundary of an image or ol the lower boundary of the image. 

One cmbodimect of toe invention uses the following mics to conceal errors in the 
missing macroblock when trocar injcrpolatioa between the upper macrobtock and the lower 
nucroblock is not applicable. 

When the missing microblock is at the upper boundary of the image, the topmost 
lino of the lower nucroblock is nsed ok IK If the lower nucroblock is also missing, the 
topmost Knc of the cexl-lowcr macroblock in the image is used as lb, and so forth, if further 
lower macrobiotics are missing. If all the lower ma deblocks arc missing, a gray line is 
used as to. 

When the missing nucroblock is at the lower boundary of the image or the lower 
nucroblock is missing, lb, the lowermost line of the upper macroblock, is also used as It 

When the mining nucroblock is neither at (he upper boundary of the image nor at 
the lower boundary of the image, and interpolation between the upper macroblock and the 
lower macroblock is net applicable, one embodiment of the invention replaces die missing 
macroblock with gray pixels (Y-U-V=128 value). 

According to one decoding standard, MPEG-4, pixels that are associated with a 
block with an error are stored as a "0," which corresponds to green pixel* in a display. 
Gray pixels can be closer than green to too colors associated with a missing block, and 
simulation tests have observed a 0.1 dB i mp ro v e m ent over the green pixels with retanvdy 
UtUo or no increase in complexity. For example, the gray pixel color can be implemented 
by a copy instruction When the spatial concealment is complete, the process ends. 

When the error relates to a predictive-coded macroblock, the second decision block 
312 determines whether another motion vector is available to be nsed for the missing 
mscroblook. For example, the video hitstream may also include another motion vector, 
such as a redundant motion vector, which can be used instead of a standard morion vector 
in the missing macrobiock. In one embodiment, a redundant motion vector is estimated by 



references motion ts the present frame at time t to a frame at time r-2. When both the frame 
at time f-2 and (he rednndsnt motion vector are availa bl e, the process proceeds from die 
second decision block 3 12 to a second state 316, where the process reco n structs the mi ss i n g 
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macrofclock from the redundant motion vector and the frame at time t-2. Otherwise, the 
process proceeds from the second decision b Jock 312 to a third decision block 320. 

In the third decision block 320, the process determines whether the error is due to & 
predictive-coded macrahlocfc missing in the present frame, Le., missing motion rectors. 
When the motion ve ctors ere missing, the process proceeds from the third decision block 
320 to a third state 324. Otherwise, the process proceeds from the third decision block 320 
to a fourth decision block 328. 

In the third state 324, the process substitutes the missing motion vectors in the 
missing rnacroblock to provide temporal concealment of the error. One embodiment of 
temporal concealment of missing motion vectors is described in greater detail later in 
connection with Figure 4. The process advances from the third state 324 to the fourth 
decision block 328. 

In the fourth decision block 328, the process determines whether an error is doe to a 
missing reference frame, o.g., the frame ct time M. If the reference frame is available, the 
process proceeds from the fourth decision block 328 to a fourth stale 332, where the 
process uses the reference frame and the substitute motion vectors from (he third state 324. 
Otherwise, the process proceeds to a fifth state 336. 

In the fifth state 336, the process uses a frame el time t-k as a reference frame. 
Where the frame corresponds to the previous-previous Came, k can equal 2. m one 
embodiment, the process multiplies the motion vectors mat were received in the 
maeroblock or substituted in the third state 324 by a factor,- such as 2 for linear motion, to 
conceal the error. The skilled practitioner will appr eciate that other appropriate, factor* may 
be used depending on the motion characteristics of the video images, lbs process proceeds 
to end until the next error u detected. 

Figure 4 illustrates an exemplary process of temporal concealment of missing 
motion vectors. In one embodiment, a maeroblock includes four motion vectors. In the 
Illustrated temporal concealment technknic^ the tmccra^ motion vectors of n rm&smg 
maeroblock 402 are substituted with motion vectors copied from other macrob locks. Io 
another embodiment, which will bo described later, the missing motion vectors of the 
missing maeroblock 402 are substituted with motion vectors interpolated from other 
mocro hl o c fcs , 
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When the missing macroblock 402 is below and above other macroblock* in the 
image, the process copies ciotxoa vectors tarn an upper macroblock 404, which is above 
the missing macro block 402. and copies morion vectors from a lower rcacroblock 406, 
which Is below the mlssmg macroblock 402. 

The missing macroblock 402 corresponds to a first missing motion vector '410, d 
second missing motion vector 412, a third missing motion vector 414, and a fourth missing 
motion vector 416. The upper macrobloefc 404 includes a first upper motion vector 420, a 
second upper motion vector 422, a third upper motion vector 424, end a fourth upper 
motion vector 426. The lower macros lock 406 includes a first lower morion vector 430, a 
second lower motion vector 432, a third lower motion vector 434, and a fourth lower 
motion vector 436. 

When both the upper macroblock 404 and the lower macroblock 406 ere available 
and include motion vectors, (he illustrated process uses the third upper motion vector 424 
os the first missing motion rector 410, the fourth upper motion vector 426 as the second 
missing motion vector 412, the first lower motion vector 430 as the third missing morion 
vector 414, and the second lower motion vector 432 as the fourth mining motion vector 
416. 

When the missing macroblock 402 at the upper boundary of the image, the process 
sets both the first missing motion vector 410 and the second missing motion vector 412 to 
the zero vector (no motion). The process uses the first lower motion vector 430 as the third 
missing motion vector 414, and the second lower motion vector 432 as the founh missing 
motion vector 4 1 6. 

When the lower macroblock 406 is corrupted or otherwise unavailable and/or fixe 
mi s sin g macroblock 402 is at the lower boundary of the image, the process seta the thin) 
missing motion vector 414 equal to tho value used for. the first missing motion vector 410, 
and the process sets the fourth missing motion vector 416 equal to the value used Cor the 
second missing motion vector 412, 

In one embodiment, the missing motion vectors of the mossing macroblock 402 arc 
substituted with motion vectors interpolated from other macroblocks. A variety of 
toenmooes for interpolation exist tn one example, the first missing motion vector 410 Is 
substituted with a vector sum of the first upper motion vector 420 and 3 times the third 
upper motion vector 424, ic, vUu " vU» + P)( v3 «*)- hi another example, the third 
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missing motion vector 414 can be substituted with a vector sum of the third lower motion 
vector 434 and 3 times flic first lower motion vector 430, Le, v3<u ■ (3)(v1 4M >+-v3tu. 

Figure 5 is a flowchart 500 generally illustrating a process or adaptrvely concealing 
errors m a video bitstrcam. Advantageously, the process of Figure S odaptively selects a 
concealment mode such that the error-concealed or reconscructed Images can correspond to 
relatively less distorted image. Simulation tests predict improvement* of up to about 1.5 
docibds (dB) in peak signal to noise ratio. The process of Figure 5 can be used to select an 
error concealment mode even when data for o present rxamo is received without an error. 

For example, the process can receive three consecutive frames. A first frame is 
cleanly received. A second frame is received with a reiatrvefy high-degree of corruption. 
Data for a third frame is cleanly received, but reconstruction of a portion of the third frame 
depends on portions of the second frame, which was received with a relatively high-degree 
of corruption. Under certain conditions, It can be advantageous to conceal portion of the 
third frame because portions of the third frame depend on a portions of a corrupted frame. 
The process illustrated In Figure 5 can advantageously identify when error wsncealment 
techniques should be invoiced even when such error concealment techniques would not be 
needed by standard video decoders to provide a display of the corresponding image. 

The process starts in a first stare 504, where the process receives data from tho video 
bhstream for the present frame, Le., the frame at time r. A portion of the received data may 
be missing, due to an error, such as a dropout, corruption delay, and the like. The process 
advances from the fust state 504 to a first decision block 506. 

m the first decision block 506, the process determines whether the dan under 
analysis corresponds to an irjlra-coded video object plane fl-VOP) or to a predictive-coded 
VOP (P-VOP> It will be understood by one of ordinary skill in the art that the process can 
operate at different levels, such as on roacroblocks or thanes, and that a VOP can be a 
frame. The process proceeds from the first decision, block 506 to a second decision block 
5 1 0 when tho VOP is on l-VOF. Otherwise, i.e>, the VOP is a P-VOP, the process proceeds 
to a (bird decision block 514. 

In the second decision block 510, the process dctemriacs whether there is an error in 
the received data for the I-VOP. Theprocess proceeds from the second decision block 510 
to a second stale 518 when there is an error. Otherwise, tho process proceeds to a rhird 
state S22. 
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la the second state 518, the process conceals the error with spatial concealment 
techniques, such as the spatial concealment techniques described earlier in connection with 
the first state 308 of Figure 3. Che process advances from the second state 518 to a fourth 
state 526. 

In the fourth state 526, the process sets an error value to an error predicted for the 
concealment technique used in the second stale 5 18. Ono embodiment normalizes the error 
to a range between 0 and 253, where 0 corresponds to no error, and 255 corresponds to a 
maximum error. For example, where gray pixels replace a pixel m an error concealment 
mode, tho error value can correspond to 255. lo one embodiment, the error value is 
retrieved from a table of prc-calculotcd error estimates, m spatiul interpolation, (he pixels 
adjacent to error-free pixels are typically more fiuthfiilly concealed than the pixels thai are 
farther away from the error-free pixels. In one embodiment, an error valne is modeled as 
97 for pixels adjacent to error-free pixels, while other pixels are modeled with an error 
value of 2 15. The error values can be maintained in a memory array en a per-ptxei basis, 
can be maintained for only a selection of pixels, can be maintained for groups of pixels, and 
so forth. 

m the third stats 522, the process bos received an error-free I-VQP and clears (to 
Zero) the error value for the cwespooding pixels of the VOP. Of course, other values can 
be arbitrarily selected to indicate an error-free stale. The process advances from the third 
slate 522 to a fifih state 530, where the process constructs the VOP from the received data 
and ends. The process can be reactivated to process tho next VOP received. 

Returning to the third decision block 514, the process determines whether the P- 
VOP includes an error. When there is an error, the process proceeds from the third decision 
block 514 to a fourth decision block 534. Otherwise, the process proceeds to an optional 
sixth state S38. 

In the fourth decision block 534, the process determines whether the error values for 
the corresponding pixels are zero or not If the error values are zero and there is no error in 
me data of the present P-VOP, then the process proeeeds to the firm state 520 and 
constructs the VOP with tho received data as this corresponds to an error-free condition. 
The process then ends or waits for the next VOP to be processed. If the error values are 
noo-zero, then the process proceeds lo a severuh state 542. 
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Id the seventh state 542, die process projects the estimate error value, 1.0., a new 
error value, that would result if the process uses the received data. For example, if a 
previous frame contained aa error, chat error may propagate to the present frame by 
decoding and using the P-VOP of the present frame. In one embodiment, the estimated 
error value is about 103 phis an error propacarioTi term, which depends on the previous 
error value The error propagation term can also include a "leaky" value, such as 0.93, to 
reflect a slight loss b error propagation per frame. The process advances from the seventh 
stale 542 to an eighth state 546. 

in the eighth state 546, the process projects the estimated error value that would 
result if the process used an error resilience technique. The error resilience technique can 
correspond to a wide variety of techniques, such as an error concealment technique 
described in connecrioo with Figures 3 and 4, the use of additional motion vectors that 
reference other frames, and the like. Where the additional motion vector references the 
previous-previous frame, one embodiment uses an error value of 46* pros the propagated 
error. It will be recognized that a propagated error hi a previous frame can be different than 
a propagated error in a previous-previous frame, m one embodiment, the process projects 
the estimated error values that would result from a plurality of error resilience techniques. 
The process advances from the eighth state 546 to a ninth state 550. 

Id the ninth state 550, the process selects between using the received data and using 
an error resilience technique. In one embodiment, the process selects between using the 
received data end using one of multiple error resilience techniques. The construction, 
concealment, or reconstruction technique that provides the lowest projected estimated error 
value is used to construct the corresponding portion of the image; The process advances 
from the ninth state 550 to a tenth state 554, where the process updates the affected error 
values according to the selected received data or error resilience technique used to generate 
the flume, rod the process ends. It win be understood that the process can then wah until 
the next VOP is received, and die process can reactivate to process the next VOP. 

hi the optional sixth stale 538, the process computes the rrojected error values with 
multiple error resilience techniques. The error resilience technique mat indicates the lowest 
projected estimated error value is selected. The process advances from the optional sixth 
state 53S to an eleventh state 558. 
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In the eleventh state 558, the process applies the error resilience technique selected 



to conceal errors far P-VOPx, the stilled practitioner will appreciate that the optional sixth 
state 538 need not he present, and the process can apply the error resilience technique ic the 
eleventh state 558 without a selection process. The process advances from the from the 
eleventh state 558 to a twelfth state 562, where the process updates the corresponding error 
values in accordance with the error resilience technique applied in the eleventh state 558. 
The process then ends and can he reactivated to process fanrre VOPs. 

Figure 6 is a flowchart 600 generally aiustrating a process that can use weighted 
predictions to compensate far errors in a video bitstream. One embodiment of the process 
is relatively less complex to implement than adaptive techniques. The illustrated process 
receives a frame of data sad processes the data one macroblock at a time. It will be 
understood that when errors in transmission arise, the process may not receive an entire 
frame of data. Rather, the process can start processing the present frame upon other 
conditions, such as determining thai (he timeframe far receiving the flame has expired, or 
reed ring data for the subsequent frame, and the like. 

The process starts in n first decision block 604, where the process determines 
whether the present frame is a prcdicti vc-coded frame fJMrame) or is an intra-coded frame 
(I-frame). The process proceeds from the first decision block 604 to a second decision 
block 60S when the present frame corresponds to an I-frame. When die present frame 
corresponds to a P* frame, (he process proceeds from the first decision block 604 to a third 
decision block 612. 

In the second decision block 60S, the process determines whether the macroblock 
under analysis includes an error. The macroblock; under analysis can correspond to the first 
macroblock" of the frame and end with the last maooblock'of the frame. However, the 
order of analysis can vary. The error can correspond to a variety of anomalies, such as 
missing data, syntax errors, checksum errors, and the Hke. The process proceeds from the 
second decision block 608 to a first state Cl6 when no error is detected in the macroblock. 
If an error is detected in the macroblock, the process proceeds to a second stale 620. 

hi the first stale 616. the process decodes the macroblock. All macroblocks of an 
intra-coded frame are intra-coded. An intra-coded macroblock can be decoded without 
reference to other macroblocks. The process advances from the first state 616 to a third 



in the optional sixth state 533. Where the process uses only 



resilience technii 
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state 624, where the process resets an error variance (EV) value corresponding to a pixel m 
the macroblock to zero. The error variance relates ta a predicted or expected amount of 
error propagation. Since the infra-coded maexobloclc does not depend on other 
macrobJockB, on error-free infra-coded maexobloclc can be expected to have on error 
variance of zero, h will be understood by one of ordinary sldll in Ihe art that any number 
can be arbitrarily selected to represent zero. It wfll also be understood that the error 
variance can be tracked in a broad variety of ways, including on a per pixel basis, on groups 
of pixels, on selected pixels, per macroblock, and the like. The process advances from the 
third state 624 to a fourth decision block 628. 

tn tha fourth decision block 628. the process determines whether h fas processed 
the lost macroblock in the frame. The process returns from the fourth decision block 628 to 
the second decision block 608 when there ore further macjohlocks in the frame- to be 
processed. When the last macroblock has been processed, the process ends and can be 
reactivated when for the subsequent frame. 

In the second state 620, the process conceals the error with spatial concealment 
techniques, such as the spatial concealment techniques described earlier in connection with 
the first slate 30S of Figure 3. In one cinbadirneni, the process fills the pixels of the 
macroblock with gray, which is encoded as 128, The process advances from the second 
state 620 to a fourth state 632, where the process sets the rnacroblock's corresponding error 
variance, o ^ , to a predetermined vahie, o ^ . In cms embodiment, the error variance, o £ , 
is normalized to a range between 0 and 255. The predetermined value can be obtained by, 
for example, simulation results, real world testing, and the like. Id addition, the 
predtfxxrnined value can depend on the concealment technique. In one embodiment, where 
the concealment technique is to fill the macroblock with gray, the predetermined 
value, a ^j- , is 255. The process advances from the fourth state 632 to the fourth decision 
block 628. 

When ths frame is a P-frame, the process proceeds from the first decision block 604 
to the third decision block 612. m the (bird decision block 612, the process determines 
whether the macroblock under analysis includes an error. The process proceeds from the 
third decision block 612 to a fifth decision block 636 when no error is detected When an 
error is detected, the process proceeds from the third decision block 612 lot fiilh state 640. 
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A macroblock in a P-fruroe can cmrespond to either an inter-coded macroblock or to 
an uztnHOxied macroblock. In Ihe fiflh decision block 636. the process drl ermines whether 
lbs macrobt ock co r re s ponds to an inter-coded xnacroblocJc or to an intro-coded macroblock. 
The process proceeds from [be fifth decision block 636 to a sixth state 644 when die 
macroblock corresponds to an mtra-coded tnacxoblock. When the macroblock corresponds 
to an inter-coded macrobloclc, tbe process proceeds to a seventh sUlo 64S, 

In the sixth state 644, the process proceeds to decode the tnira-coded raacrobfock 
that was received without an error. The intro-coded macroblock can be decoded without 
reference to another macroblock. Tbe process advances from the sixth state 644 to an 
eighth slate 652, where the process resets (he corresponding error variances maintained for 
the macroblock to zero. The process advances from tbe eighth state 652 to a sixth decision 
block 664. 

In tbe sixth decision block 664, the process determines whether it has processed the 
last macroblock in the frame. The process returns from the sixth decision block 664 to the 
third decision block 612 when there are farther njanroblocks in tbe frame to be processed. 
When the Inst macroblock has been processed, the process ends and can be reactivated far 
the subsequent frame. 

In the seventh state 648, the process reamstrocts the pixels of the macroblock even 
when the macroblock was received without error. Reconstruction in this circumstance can 
improve image quality because a prevLoos-prcrious frame may exhibit less corruption than 
a previous-frame. One embodiment of the process selects between a first i hxjo struct i on 
mode and a second reconstruction mode depending on which mode is expected to provide 
better error concealment- In another embodiment, weighted sums are used to combine tbe 
two modes. In one example, the weights used correspond (o tbe inverse of estimated errors 
so that the process decodes with minimal mean squared error (MMSE). 

m the first reconstruction mode, the process reconstructs the macroblock based on 
the received motion vector and be corresponding portion in the previous frame. The 
reconstructed pixel, q t , as reconstructed by the first reconstruction mode, is expressed in 
Equation 1. In Equation 1, f> t is a prediction residual 

m fee second reconstruction ruodo, tbe process reconstructs the macroblock by 
doubling the amount of motion specified by me rae^ vectors erf m^ 
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process uses a corresponding portion of (he previous-previous game, i*, (he fame at time 

The error variance of a pixel reconstructed by the first reconstruction mode, o^.is 
expressed in Equation 2, where k indicates the frame, e.g, A-0 for (ho present (rams. The 
error variance of a pixel reconstructed by tho second reconstruction mode, is 
expressed in Equation 3. 

*L, -W) (Eq.2) 

Id one embodiment, the process selects tho second reconstruction mode when 
<r^ In another embodiment, weighted sums are used to combine the 

reconstruction tec hniq ues. In one example, the weights used correspond to the inverse of 
predicted errors so that the process decodes with minimal mean squared error (MMSB). 
With weighted sums, the process combines die two predictions to reconstruct the pixel, q*. 
m one embodiment, ftepix dp* is reenns^ q k , aa expressed in Eqantion 4. 

£ = ~&)P4- 3 +r, (Eq-4) 

In one embodiment, the weighting coefficient, fi t is calculated from Equation 5, 

The process advances Horn me seventh state 648 to a ninth state 656. hi tho ninth 
state 656, the process updates the corresponding error variances for the macroblocx based 
on the reconstruction applied in the seventh state 64S. Tho process advances from the from 
the ninth state 656 to the sixth dtrtKiwi block 664. In one embodiment, the error variance 
is calculated from expression in Equation 6. 

"fa*"****** 

In the fifth state 640, the process conceah mo errors in the macrobloox. A variety of 
concealment techniques can be applied. In one embodiment, the process uses temporal 
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concealment regardless of whether the maeroblock is mtra-codisJ or inter-coded It will be 
understood Out in other embodiments, the type of coding nsed in the maeroblock can be 
used as & factor in the selection of a concealment technique. 

One embodiment of the process selects between a first conceabnenr mode based on 
a previous frame and a second concealment mode based on a previous-previous fame in 
the Gfih state 640. In the first concealment mode, the process generates an inter-coded 
maeroblock for the missing maeroblock using the motion vectors extracted from a 
maeroblock that is above the missing maeroblock m the image. If the maeroblock that is 
above the missing maeroblock has an error, the motion vectors can be set to zero vectors. 
The corresponding portion of the frame is reconstructed with the generated inter-coded 
maeroblock and the corresponding reference information from the previous frame, i.e., the 
frame at 

In the second concealment mode, the process generates an mier-eoded maeroblock 
for the missing maeroblock by copying and multiplying by 2 the motion vectors extracted 
from a maeroblock that is above the missing maeroblock in the image. If the maeroblock 
above the missing maeroblock has an error, the motion vectors can be set to zero vectors. 
The corresponding portion of the frame is reconstructed with the generated inter-coded 
maeroblock and the corresponding reference information from the previous-previon* frame, 
tx, the frame at 1*2. 

The error variance can be modeled as a sum of the associated propagation error and 
concealment error, it one embodiment, iho first concealment mode has a lower 
concealment error than the second ccmceaunent mode, but the second concealment mode 
has a lower propagation error than the first concealment mode. 

In one embodiment, the process selects between the first concealment mode and the 
second concealment mode based on which one provides a lower estimated error variance. 
In another embodiment, weighted sums are used to combine the two modes, m Equation 7, 
cr^ 0 , denotes the error variance of a pixel q t . The value of / is equal to 1 for the first 
concealment mode based on the previous frame and is equal to 2 for the second 
concealment mode based oo the previous-previous frame. 
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In Equation 7, a corresponds to the error variance for the concealment mode 
and o ^ corresponds Co the propagation error variance. 

In another embodiment, the process computes weighted sums to farther reduce the 
error variance of the concealment For example, q t can be replaced by q k as shown in 
Eduction 8. 

q k « or M + (1 -a)cV, (Eq. S) 
m one embodiment, the weighting coefficient, a, is as expressed in Equation 9. 

The process advances from the filth state to a tenth state 660. In the tenth state 660, 
the process updates the corresponding error variances for the macroblock based on the 
concealment applied in the fifth, stale 640, and (he process advances to the sixth decision 
block 664. In one embodhnent with weighted sums, the error variance is calculated from 
expression in Equation 1 0. 

K - m k -?»>'> =4^4*2- a*. io) 

a lt0 +O *P> 

In some situations, an entire frame is dropped or lost One embodiment of the 
invention advantageously repeats the previous frame* or interpolates between the previous 
frame ant) the next frame, in response to a detection of a frame that is cursing from a frame 
sequence. In a real-time application, the display of the sequence of frames can be slightly 
delayed to allow the decoder time to receive the next frame, to decode the next frame, and 
io generate the interpolated replacement frame from the previous frame and the next frame. 
The missing frame can be detected by calculating a frame rate from received frames and by 
caJcnlaling an expected time to receive a subsequent frame. When a frame does not arrive 
at the expected time, it is replaced with the previous frame or rrrterpolaied from the 
previous and next frames. One embodiment of the process further resynchrorhzes (he 
available audio portion to correspond with the displayed images. 

Data corruption is an occasionally unavoidable occurrence. Various techniques can 
help conceal errors in the transmission or reception of video data. However, standard video 
decoding techniques can inefficiently declare error-free data as erroneous. For example, the 
MPEG-4 standard recommends dumping an entire macrocJocic when an error is detected fca 

-28- 



(81) 



JP 2004-528752 A 2004.9.16 



WO W,l 736 PCT/USOi'«6«5 

the macroblock. The fMlowing lecruriqucs illustrate that data Car some macroblocks can be 
reliably recovered and used from video packers with corruption. For example, a 
maeroblocfc in an MPEG-4 system can contain six 8-by-S imago blocks-. Four of the image 
blocks encode luminosity, and two of the image blocks encode chroraaticity. In one 
conventional system, all six of the image blocks are .discarded even if a transmission error 
wens only to affect one image black. 

Figures 7A and 7B illustrate sample video packets. In an MPEG-4 system, video 
packets include rcsyncbrocizalion markers to indicate the start of a video packet The 
number of macroblocks within a video packet can vary. 

Figure 7A illustrates a sample of a video packet 700 with DC and AC components 
for an I-VOP. The video packet 700 includes a video packet header 702, which includes 
toe resyiKhromzatioo marker and other header information that can be used lo decode the 
macroblocks of the packet, such as the macroblock number of the first mscrofalock hi the 
packet and the quantization parameter (QP) to decode (he packet, A DC poctkm 704 can 
include mebpc, depuuu, and dc data, sucb as hmrinoshy. A DC marker 706 separates the 
DC portion 704 from an AC portion 708. In one embodiment the DC marker 70S is a 19- 
bit binary string **1 1 0 1011 0000 0000 0001.** The AC portion 70S can include an acjpred 
flag and other textual information. 

Figure 7B illustrates a video packet 720 for a P-VOP. The video packet 720 
includes a video packet header 722 similar to the video packet header 702 of Figure 7 A 
The video packet 720 further includes a motion vector portion 724, which includes motion 
dat a. A motion marker 726 separata the motion dala in the motion vector portion 724 
from texture data in a DCT portion 728. In one embodiment, the motion marker is a 17-bit 
bhuny string "I 1111000000000001." 

Figure S niustrates an example of discarding a corrupted macroblock. Reversible 
variable length codes (RVLC) are designed to allow data, such as texture codes, to be read 
or decoded in both a forward direction 802 and a reverse or backward direction 804. For 
example, in the forward direction £02 with N rnacrcb locks, a first rjiaeroblock S06, MB #0, 
is read first and a last macroblock SOS, MB if is read last An error can be located in a 
macroblock S10, which can be used to define a range of macroblocks S12 that are 
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Figure 9 is a flowchart thai generally illustrates u process according to an 
embodiment of the invention of partial RVLC decoding of discrete cosine transform (DCT) 
portions of corrupted packets. The process starts at a first state 904 by reading macroblock 
iirtmorion, such as the macroblock number, of the video packet header of the video 
packet The process advances from the first stale 904 to a second Rate 903. 

In the second state 90S, the process inspects the DC portion or the motion vector . 
portion of the video pacbet, as applicable, lite process applies syntactic and logic tests to 
the video packet header and to tho DC portion ox motion vector portion to detect errors 
therein. The process advances from the second state 90S to a first decision block 912. 

In the first decision block 912. the exemplary process determines whether there was 
an error in the video packet header from the first state 904 or the DC portion or motion 
vector portion from the second state 90S. The first decision block 912 proceeds to a third 
state 916 when the error is detected. When the error is not detected, the process proceeds 
from the first decision block 912 to a fourth state 920. 

In the third state 916, the process cUscards tbe vidteo packet. It will be understood by 
one of ordinary skill in the art that errors in the video packet header or in (he DC portion or 
motion vector portion can lead to relatively severe errors if incorrectly decoded. In one 
embodiment, error concealment techniques are instead invoked, and the process ends. The 
process can be reactivated later to read another video packet 

In the fourth state 920, tho process decodes the video packet in tho forward 
direction. In one embodiment, the process decodes the video packet according to standard 
MPEG-4 RVLC decoding techniques. One embodiment of the process main tarns a count of 
macroblocks in a macroblocks counter. The header at the beginning of the video packet 
includes a macroblock index, which can be used to initialize the rnacroblocks counter, As 
decoding proceeds in the forward direction, the macroblock counter increments. When an 
error is encountered, one embodiment removes one count from tho macroblocks counter 
such that the macroblock counter contains the number of completely decoded macroblocks. 

In addition, one embodiment of the process stores all codewords as leaves of a 
binary tree. Branches of the binary tree arc labeled with either a 0 or a 1 . One embodiment 
of the process uses two different tree formats depending on whether the macroblock is intra 
or inter coded. When decoding in the forward direction, bits from the video packet are 
retrieved from a bit buffer containing the RVLC data, and the process traverses the data in 
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the tree until one of 3 events is encountered. These events correspond to a first event where 
8 valid codeword is reached al a leaf-nodc; a second event where m invalid leaf of the 



the end of the bit buffer is reached. 

The first event indicates no error. With no error, a valid RVLC codeword is 
mapped, such as via a srrapJe lookup table, to its corresponding leaf-node {last, run, fcve/L 
la one embodiment, (his information is stored in an stray. When an entire S-by-8 block is 
decoded, as indicated by tho presence of ao RVLC codeword with tort-), the process 
proceeds to decode the next block until an error is encountered or the last block is leached. 

The second event and the third event correspond to errors. These errors can be 
caused by a variety of error conditions. Examples of error conditions include an invalid 
RVLC codeword, such as wrong marker bits in the expected locations of ESCAPE 
symbols; decoded codeword from an ESCAPE symbol results in (run, length, level) 
information that should have been encoded by a regular (non-ESC APE) symbol; more than 
64(or63 Ibr the case of Ictra-blocks with DC coded separately from AC) DCT coefficients 
b an S-by-S Mock; extra bits remaining after successfully decoding all expected DCT 
coefficient; of all 8-by-g blocks in a video packet; and insufficient bits to decode oil 
expected 8-by-S blocks in video packet These conditoiui can be tested secn^nlially. For 
example, when testing for extra bits remaining, the condition is tested afler all the S-by-8 
blocks in the video packet are processed In onoiher example, the testing of the manner of 
DCT coefficients can bo performed on a block-by-block basis. The process advances from 
the fourth state 920 to a second decision block 924. However, it will be nrjcxrstood by the 
stilled practitioner that (he fourth state 920 and the second decision block 924 can be 
included in a loop, such as a FOR loop. 

In the second decision block 924, the process determines whether there has been an 
error in the forward decoding of the video packet as described in die fourth state 920 (in the 
forward direction). The process proceeds from the second decision block 924 to a fiflh 
stale 928 when there is no error. If there is to error in the forward decoding, the process 
proceeds from the second decision block 924 to a sixth state 932 and co a teeth state 945. 
Upon an error in forward decoding, the process terminates further forward decoding and 
records the error location and type of error in the tenth state 948. The error location in tho 



binary nee (not 



to any RVLC codeword) is reached; and a third event where 
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forward direction, L\, and the number of completely decoded macron locks in (be forward 
direction, N\, win be described in greater detail later in connection with Figure* 10-13. 

m the fifth state 928, the process reconstructs the DCT coefficient blocks and ends. 
In one embodiment, the reconstruction proceeds sccordmg to standard MPEG-4 techniques. 
It will be understood by one of ordinary skin in the azt that the process can be reactivated (u 
proccij the next video packet. 

]n the sixth stale 932, (he process loads tite video packet data to a bit buffer. In 
order to perform partial RVLC decoding, detection of the DC (for I-VOP) or Motion (for P- 
VOP) markers for each video packet should be obtained without prior syntax errors or dan 
overrun. In one embodiment, a circular buffer that reads data for the entire packet is used 
lo obtain the remaining bits for a video packet by unpacking each byte to 8 bits. 

The process removes starting bits from the end of the buffer, which leaves only data 
bits in the RVLC buffer. During parsing of the video packet header end motion vector 
portion or DC portion of the video packet, the expected number of manohlodu, the type of 
each one macroblock (INTRA or INTER), whether a micro block is slapped or not, how 
many and which of the expected 4 luminance and 2 chrominance S-by-8 blocks have been 
coded and should thus bo present in the bitstream, and w he th er INTRA blocks have 63 or 
64 coefficients fi.c whether their DC coefficient is coded together or separate from the AC 
coefficients) should be known. This information can be stored in a data structure with the 
RVLC data bits. The process Advances from the sixth state 932 (o a seventh slate 936. 

In the seventh state 936, the process performs reversible variable length cods 
(RVLC) decoding in the backward direction on (he video packet. In one embodiment, the 
process performs the backward decoding on the video packet according lo standard MPEG- 
4 RVLC decoding techniques. The inaadmmn number of d e coded codewords should be 
recovered in each direction. One embodiment of the process maintains the number of 
completely decoded nmcroblocks encountered in the reverse direction in a counter. In one 
embodiment, the counter is banalized with a value from (he video packet thai relates 
to the number of macroblocks expected in the video packet. A", and (he counter counts down 
as macroblocks are read. The process advances from the seventh state 936 to an eighth 
state 940. 

A the eighth state 940, the process detects an error In (he video packet from the 
baclcward d eco din g and records the error and the type of error. In addition to the errors for 
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the forward direction described earlier in connection with the fourth state 920, another error 
thai can occur in the reverse decoding direction occurs when the last decoded codeword, 
Lc, the first codeword in the reverse direction, decodes to a codeword with la$t=0. 
Advantageously, detection of the location of the error in the reverse direction can reveal 
ranges of data where such data Is still usable. Use of the error location in the reverse or 
backward direction, L>> and use or the number of completely decoded macroblocks in (be 
reverse direction, M, will be described later in connection with Figures 10*13. 

m the exemplary process, different decoding trees (INTRA/INTER) are used for 
reverse decoding direction than in the forward decoding direction. Id one embodiment, the 
reverse decoding trees are obtained by reversing the order of bits for each codeword. In 
addition, one embodiment modifies (be symbol decoding routine to take into account that a 
sign bit thai is coming last in forward decoding is encotmlered first in backward decoding; 
and thai LasvX indicates the last codeword of on 8-by^ block in forward decoding, but 
indicates the first codeword in reverse decoding. When decoding hi the reverse direction, 
the very first codeword should have fcwr-1 or otherwise an error is declared. 

When data is read in the reverse order, the process looks ahead by one symbol when 
decoding a block. If a codeword with Lur°[ is reached, the process has reached the end of 
reverse decoding of the current S-by-8 block, and the process advances to the next block. 
In addition, the order of the blocks is reversed for the same reason. For example, if 5 
INTER blocks followed by 3 INTRA blocks are expected in the forward direction, 3 
INTRA blocks followed by 5 INTER blocks should be expected in the reverse direction. 
The process advances from the eighth state 940 to a ninth stale 944. 

In the ninth state 944* the process discards overlapping error regions from the 
forward and the reverse decoding directions. The 2 arrays of decoded symbols are 
compared to evaluate overlap in error between the error obtained daring for war d RVLC 
decoding and the error obtained during reverse RVLC decoding to partially decode the 
video packet Further details of partial decoding will be described in greater detail later in 
connection with Figure 10-13. It wjJl be imdcrstood by one cf otdmary iloll in the art that 
that in the process described herein, the arrays contain the srxcessrhJry decoded codewords 
before any decoding error has been declared in each direction. If them is no overlap 
between successfully d ec od ed regions in forward and reverse direction at the bit-level and 
also at the DCT (Macroblock) level, then one embodiment performs a cortservative 

-33- 



(86) 



wo 02*71 7* rcT/usoaoww 

backtracking of a predetermined number orbits, T, such as about SO bhs to each direction, 
La. the last 90 bits in cadi direction are discarded. Those codewords thai overlap (ra the 
bit buffer) or decode to DCT coefficients that overlap (in the DCT buffer) etc discarded, m 
addition, one embodiment retains only entire INTER mocroblocks (no partial xnacroblock 
DCT data or Intra -coded macroblocks) in the decoding buffers. The remaining codewords 
are then used to reconstruct the S-by-8 DCT values for individual blocks, and the process 
ends. It will be understood that the process can be reactivated to process the next video 
packet 

The process ilmstraied in Figure 9 reveab the location of the error (the bit location) 
in the forward direction, Z t ; the location of (be error in the reverse direction, Zj; (he type of 
error that was encountered in the forward direction and in the reverse direction; (tie 
expected length of the video packet, L; the mtmher of expected macroblocks in the video 
packet, jV. the number of completely decoded macroblocks in the forward direction, and 
the number of completely decoded macroblocks in the reverse direction, rV> 

Figures 10-13 illustrate partial RVLC decoding strategies. In one exemplary partial 
RVLC decoding process, a partial decoding strategy for extraction of useful data from a 
video packet is selected according to one of four outcomes. Processing of a first outcome, 
where I, +L* < A and A r , + N, < N, win be described later in connection with Figure 10. 
Processing of a second outcome, where L x + Lz < L, and + .Vj >= N, will be described 
later m connection \vim Figure 11. Processing of a third outcome, whereZi +Zj>-Z, and 
N t + »Vj < Jv", will be described later in connection with Figure 12. Processing of a fourth 
outcome, where Zi + Z 2 >»Z, and M + N, >- N> wiU be described later in connection with 
Figure 13. 

Figure 10 ilhatrales a partial decoding strategy used when Zi + Zj < L, and N t + % 
<N. A Gnu portion 1002 of Figure 10 indicates the hit error positions, L\ and Zj. A second 
portion 1004 indicates the completely decoded macroblocks in the forward direction, N iy 
and in the reverse direction, rY a . A third portion 1006 indicates a backtracking of tuts, T, 
from (bo hit error locations. It wiU be Understood by one of ordinary skill in (he an that the 
number selected for the backtracking of bits, T, can vary in a very broad range and can even 
be different in Uie forward direction and in the reverse direction, ti one crnbodiment, the 
value of Tis 90 bits. 
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The exemplary process apportions the video packet in a first partial picket 1010. a ' 
second partial packet 1012, and a discarded partial packet 1014. The first parted packet 
1010 may be used by the decoder and includes complete macroblocks up to a bit position 
corresponding to L\*T. The second partial packet 1012 nay also be used by the decoder 
end includes complete macrobbeks from a bit position corresponding to L-L\+T to the end 
of the psrkcl, L, such that Ote second partial packet is about Lj-T in size. As described in 
greater detail later in connection with Figure 14, one embodiment of the process discards 
intra blocks in the first partis] packet 1010 and in the second partial packet 1012, even if the 
intra blocks are identified as nncorrupted. The discarded partial packet 1014, which 
includes the remaining portion of the video packet, is discarded. 

Figure 11 fflnstrotes a partial decoding strategy used when Zi +1»<Z, and/* 
>= N. A first portion 1 102 of Figure 1 1 indicates the bit error positions, L\ and 1*. A 
second port ton 1104 indicates the completely decoded macroblocks in the forward 
direction, N u and in the reverse direction, tf 2 - 

The exemplary process apportions the video packet in a first partial packet 1 110, a 
second partial packet 1 1 12, and a discarded partial packet 11 14: The first partial packet 
1 1 10 may bo used by the decoder and includes complete macroblocks from the start of the 
video packet to the maoroblock corresponding to N-N r I . The second partial packet 1112 
may also be osed by the decoder and includes the (A r i+l)th macroblock to the last 
macroblock in the video packet,- such that the second partial packet 1 1 12 is about WVV 1 in 
size. One embodiment of the process discards intra blocks in the first partial packet 1H0 
rod in the second partial packet 1 1 12, even if the intra blocks arc identified as tnxctfrupted. 
The discarded partial packet 1114, which includes Ok remaining portion of the video 
packet, is discarded. 

Figure 12 illustrates a partial decoding strategy used when L\ + >- Z, and M 4 
N) <N. A first portion 1202 of Figure 12 indicates the bit error positions, Z» and Z,. A 
second portion 1204 Indicates the completely decoded macroblocks in the forward 
direction, N h and in (he reverse direction, AY 

The exemplary process apportioas (he video packet in a first partial packet 1210, a 
second partial packet 1212, and a discarded partial packet 1214. The first partial packet 
1210 may be used by the decoder and includes complete macroblocks from the beginning 
of the video packet to a macroblock at /V-b_mbCii), where bmbfa) denotes the 
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macroblock at the bit position L v The second partial packet 1212 may also be used by the 
decoder and mchides the complete macroblocks from (he hit position corresponding to U to 
the end of the packet One embodiment of the process discards intra blocks in the first 
partial packet 1210 and in the second partial packet 1212, even if the intra blocks ore 
identified as uncorrupted The discarded partial packet 12 K which tnelodes the renaming 
portico of the video packet, if discarded. 

Figure 13 illustrates a partial decoding Strategy used wben £, + Li >-L, sod /V t *■ N 2 
>« N. A first portion 1302 of Figure 13 indicates the bit error positions, Li and L 2 . A 
second portion 1304 indicates the completely decoded mocroblocks in the forward 
direction, A/j» end in the reverse direction, W«. 

The exemplary process apportions the video packet in a first partial packet 1310. a 
second partial packet 1312, and a discarded partial packet 1314. The first partial packet 
13 10 may be used by the decoder and includes complete macroblooks up to the bit position 
corresponding to Ihe lesser of A^bjrnbCLj). where b_mb{I 1 ) denotes the last complete 
macroblock op to bit position L 2 , and the complete macroblocks up to (AWj-I)th 
macroblock. The second partial packet 1312 may also be used by tha decoder and includes 
the number of complete macroblocks counting from the end of the video packet 
corresponding to the lesser of JV-fmbfo), where fmb(Z,) denotes the last macroblock in 
me reverse direction that is unco crop ted as determined by the forward direction, and the 
number of complete macroblocks conespooding to AMft-1. One embodiment of the 
process discards intra blocks in the first partial packet 1310 and mtbe second partial packet 
1312, even if the intra blocks are identified as uncorrupted The discarded partial packet 
1314, which includes the remaining portion of the video packet, is discarded. 

Figure 14 illustrates a partially corrupted video packet 1402 with si least one intra- 
coded macroblock. In one embc<timem. an htfra-coded maccobloct in a portion of a 
partially corrupted video packet is discarded even if the fatra-coded macroblock is in a 
portion of the partially corrupted video packet (hat is considered iincerrtrpted. 

A decoding process, such as the process described in connection with Figures 9 to 
13, allocates the partially corrtrpted video packet 1402 to a first partial packet 1404, a 
corrupted partial packet 1406, and a second partial packet 1408. The first partial packet 
1404 and the second partial packet 1408 are considered error-tree and can be used. The 
corrupted partial packet 1406 tnelodes corrupted darn and should not be used 
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However, the i Dm baled first partial pocket 1404 includes a first wtnvcoded 
macroblock 1410, and the illustrated second partial packet 1408 includes a second intra- 
coded macroblock 1412. One process according to an embodiment of (he mvenrjon also 
discards an infra-coded macroblock, sach as the first mtrs-coded nwc rob Jock 1410 or tbe 
second intra-coded macroblock 1412, when any error or corruption is detected in the video 
packet, and the process advantageously continues to use the recovered macroblocks 
corresponding to error-free macroblocks. Instead, the process conceals the intra-coded 
macroblocks of the partially corrupted video packets. 

One embodiment of the invention partially decodes tarn-coded macroblocks from 
partially corrupted packets. According to (he MPEG -4 standard, any data from a corrupted 
video packet is dropped. Intra-coded niacrobJocks can be encoded in both 1-VOPs and in P- 
VOPs. As provided in tbe MPEG-4 standard, a DC coefficient of an intra-coded 
macroblock and/or the top-row and first-cohnrm AC coefficient of the intra-coded 
macroblock can be predictively coded from the intra-coded macro block's neighboring intra- 
coded macroblocks. 

Parameters encoded in the video bhstream can hiMente- the flntm pri yte mode of 
operation. A first parameter, referred to in MPEG-4 as *1nira_de_vlo_thr," is located in (he 
VO? header. As set forth in MPBQ-4. the first parameter, intra_dc_vlc_thr, is encoded to 
one of 8 codes as described in Table L where QP indicates a quantization parameter. 



Index 


Meaning 


Code 


0 


Use Intra DC VLC for entire VOP 


000 


1 


Switch to Intra AC VLC at running QP>=13 


001 


2 


Switch to Intra AC VLC at running QP>°15 


010 


3 


Switch to Intra AC VLC at running QP>~1 7 


on 


4 


Switch to Intra AC VLC at running QP>=19 


100 


5 


Switch to Intra AC VLC at running QP>-21 


101 


6 


Switch to bran AC VLC at running QP>-23 


no 


7 


Use mtra AC VLC for entire VOP 


in 



Tabic I 

The intra w dc.vlc_tfcr code of "000" corresponds to separating DC coefficients from 
AC coefficients in mtra-coded macroblocks. With respect to an I- VOP, the setting of the 
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intra_dc_v!c_thr paiamcter to "000" result* in the pbcemeat by the encoder of (he DC 
coefficient before the DC mar tar, and the placement of (he AC coefficients after the DC 
marker. 

With respect to a P-VOP, the setting of the forrajle_vic_thr parameter to "000" 
results in the encoder placing the DC coefficients immediately after the motion marker, 
together with the copy and ac_pred_flzg information. [1 will be understood thai ihe value of 
the imra_de_vlc_thr parameter is selected at the encoding level For error resilience, video 
bhsrreams maybe relatively more robustly encoded with the U3tra_dc_v1c_thr parameter set 
to 000. Nonetheless, one embodiment of the invention advantageously detects the setting 
of the intia_dc_vlc_thr parameter to "000," and monitors for the motion marker and/or the 
DC marker, (f (he corresponding motion marker and/ox is observed without an error, the 
process classifies the DC information received ahead of the morion rnartccr and/or DC 
marker and uses the DC mformarkm in decoding. Otherwise, the DC information Is 
dropped. 

A second parameter, referred to in MFEG-4 as ^acjjred^flag" is located after the 
motion marker / DC marker, bar before RVLC texture data. The "ac_pred_fUg" parameter 
instructs the encoder to differentially encode and the decoder to differentially decode the 
top row and first column of DCT coefficients (a total of 14 coefficients) from a neighboring 
block that has the best match with the current block with regard to DC coefficients. The 
neighboring block with the smallest difference is used as a prediction block as shown in 
Figure 15. 

Figure IS illustrates a sequence of macrohlocks with AC prediction. Figure 13 
includes a first niacrohlock 1302, A. a second macroblock 1504. B, a third rnacrobJock 
1506, C, a fourth macroblock 1508, D, a fifth macroblock 1510, X, and a sixth macroblock 
1512, Y. The fifth macroblock 1510, X, and the sixth macroblock 1512, Y, are encoded 
with AC prediction enabled. A first column of DCT coefficients from the first rnacrobJock 
1502. A, is used in the fifth macroblock 1510, X, and the sixth macroblock 1512, Y. The 
top row of coefficients from the third macroblock 1506, C, or from the fourth macroblock 
1508, D, ia used to encode the top row of the fifth macroblock 1510, X, or the sixth 
macroblock 1512, Y, respectively. 

It will be understood that for error resilience, the encoder should disable the AC 
prediction or differential encoding for intra-coded macreblocka. With me AC prediction 
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disabled, infra-coded macrob locks that correspond to either the firm or second "good" part 
of the RVLC data can be used 

In one embodiment, with AC prediction enabled, the infra-coded macrob locks of the 
"good" pan of the RVLC data can bo dropped as described earlier in connection with 
Figure 14. 

In addition, one decoder or decoding process according to an embodiment of the 
invention further determines whether the uitra-coded macrob lock, referred to os "suspect 
infra-coded macroblook" can bo used even with AC prediction enabled. The decoder 
determines whether another irttra-coded raaerablook exists to the immediate- left or 
immediately above the suspect intm-coded macrob lock. When no such oiber infra-coded 
macroblock exists, the suspect infra-coded macro-block is labeled "good," and is decoded 
and used. 

One decoder further determine* whether any of (he other inacroMocks to the 
immediate left or immediately abova the suspect infra-coded macroblock have nor been 
decoded. If mere are any such macroblocks, lha suspect intm-coded macroblock is not 
used. 

Figure 16 iUusfrsies a bit structure for an MPEG-4 data partitioning packet. Data 
partitioning is an option that can be selected by the encoder. The data partitioning packet 
includes a resync marker 1502, a macroblockjiurabcr 1604, a quam_seale 1606, a header 
extension code (HEC) 1 608, a motion and header mformatian 1610, amotion marker 1612, 
a texture information 1614, and a resync marker 1 616. 

ThoMPEG-4 standard allows the DC portion of frame data to be placed in the data 
partitioning packet either before or after the AC portion of frame data. The order is 
determined by the encoder. When data partitioning is enabled, the encoder includes motion, 
vectors together with '"not-coded* 4 and "mcbpo" information in the motion and header 
information 16 10 ah ea d of the motion marker 1612 as part of header information as shown 
in Figure 16. 

When En error is detected in the receiving of a packet, bat the error occurs after the 
motion marker 1612, one embodiment of the invention uses the data received ahead of the 
motion marker 1612. One artbcdimcrLl predicts a location for the motion marker 161 2 and 
detects an error based on whether or not the motion marker 1612 was observed in the 
predicted location. Depending on the nature of the scenes encoded, the data included in tho 
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motion and bender information 1610 can yield a wealth amount of information that can be 
advantageously recovered. 

For example, when the "not cod ad" flag is set, a macroblock should be copied from 
the same location in tho previous frame by the decoder. The macroblocks corresponding to 
The "'not coding" flag can be reconstructed safely. The ^mcbpc" identifies winch of the 6 8- 
by-8 blocks that form a macroblock (4 for luminance and 2 fur chrominance) have been 
coded and. thus include corresponding DCT coefficients in the texture information 1 61 4. 

When RVLC is enabled, tho texture information 1614 is further divided into a first 
portion and a second portion. The feat portion immediately following the motion marker 
1612 includes "cbpy" mfbrmaiion, which identifies which of the 4 luminance S-by-8 blocks 
. are actually coded and which are not. The cbpy information also includes a DC coefficient 
for those imra-eoded irtacrcWocks in the packet for which the corrcspondmg "Intra DC 
VXC encoding" has been enabled. 

The cbpy information further includes an acjpred_fiag, which indicates whether the 
corresponding tutra-coded macroblocks have been cHfTercniially encoded with AC 
prediction by the encoder from other macroblocks that are lo the immediate left or are 
iramecHaxcly above the inneroblocfc. In one embodiment, the decoder uses all of or a 
selection of the cbpy information, the DC coefficient, and the ac_pred_tlag in conjunction 
with the presence or absence of a first error-free portion of the DCT data in the texture 
in formation 1614 to assess which part can be safely decoded. In one example, the presence 
of soch a good portion of data indicates that DC coefficients of intra macroblocks and cbpy- 
inferred non-coded Y-b locks of a macroblock can be decoded. 

Ons technique used in tfgital communications lo increase the robustness of 
transmitted or stored digital information is forward error correction (FEC) coding. FEC 
coding includes the addition of error corr e cti on information before data is stored or 
transmitted. Part of (he FEC process can also include other techniques such as bit- 
interieaving, Bod the original data and the error correction information axe stored or 
transmitted, and when data is lost, the EEC decoder can reconstruct the missing data from 
the data that it received and the cttot correction information. 



Advantageously, enibodimems of the irrvcntiOTi decode FEC codes in an efficient 




r. One drawback to FEC ceding technique; is that the 
cases the amount of data that is stored or transmitted, 
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referred to os overhead. Figure 17 illustrates one example of a tradeoff between block cnor 
rale (BER) correction capability versus overhead A horizontal axis 1710 corresponds tn an 
average BER correction capability. A vertical axis 1720 corresponds to ao amount of 
overhead, expressed In Figure 1? in percentage. A first curve 1730 c orr e sp onds to a 
theoretical bit overhead versus BBR correction capability. A second curve 1740 
corresponds to one example of au actual example of overhead versos BER correction 
capability. Despite the overhead costs, (he benefits of receiving the original data as 
intended can outweigh the drawbacks of increased data storage or transmission, or the 
drawbacks of a revised bit allocation in a bandwidth limited system. 

Another disadvantage to FEC coding is that the data, as encoded with FEC codes, 
may no longer be compatible with systems and/or standards in use prior to FEC coding. 
Thus, FEC coding is relatively difficult to add to existing systems and/or standards, such as 
MPEG-4. 

To be compatible with existing systems, a video bitstream should be compliant with 
a standard syntax, such as MPEG-4 syntax. To retain compatibility with existing systems, 
embodiments of the invention advantageously decode FEC coded bi (streams that are 
encoded only with systematic FEC codes and not non-systematic codes, and retrieve FEC 
codes from identified user data video packets. 

Figure 18 illustrates a video bitstrcam with systematic FEC data. FEC codes can 
correspond to either systematic codes or non-systematic codes. A systematic code leaves 
(he original data untouched and appends the FEC codes separately. For example, a 
conventional bitstxeam can include a first data 1810, a second data 1830, and so forth. 
With systematic coding, the original data, ie„ the first data 1810 arid the second data 1830, 
is preserved, and the FEC codes are provided separately. An example of the separate FEC 
code is illustrated by a first FEC code 1S20 and a second FEC code 1840 in Figure 18. In 
One embodiment, the data is carried in a VOP packet, and the FEC codes are carried in a 
user data packet, which follows the corresponding VOP packet in the bitstrcam. One 
embodiment of the encoder includes a packd of FEC codes in a user data video packet for 
eoch VOP packet However, it will be understood that depending on decisions made by the 
encoder, less than every corresponding data may be supplemented with FEC codes. 

By contrast, in a non-systematic code, the original data and the FEC codes are 
combined. It will be understood by one of ordinary skill in the art that the application of 
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FEC techniques thai generate nOD-systanatic code result in bitalrcams should be avoided 
where (he applicable video standard does not specify FEC coding. 

A wide vinery of FEC coding types can bo used. In one embodiment, tho FEC 
coding techniques correspond to Bose-Chaudhuri -Hocqueoghem (BCH) coding teenniqoes. 
In one cmbodinjcnt, a block size of 3D is used. In the illustrated configurations, the FEC 
codes are applied at the packetizer level, as opposed to another level such as a channel 
level 

In the context of an MPE&4 system, one way of including the separate systematic 
error correction data, as shown by the first FEC coda 1820 and the second FEC code 1840, 
is to include the error correction data in a user data video packet. The user data video 
packet can bo ignored by a standard MFE&4 decoder. In tbe MPECM syntax, a data 
packet is identified as a user data video packet in the video bitstrcarn by a user data start 
code, which is a bit string of 00000132 in hexadecimal (start code value of B2). as (he start 
code of the data packet Various data can be included with tbe FEC codes in the user data 
video packet, hi one embodiment, a user data header code identifies the type of data in the 
user data video packet. For example, a 16-bit code for the user data header code can 
identify that data m the user data video packet is FEC code. In another example, such as in 
a standard yet to bo defined, the FEC codes of selected data are carried in a dedicated data 
packet with a unique start code. 

it will be appreciated that error correction codes corresponding to all the data in the 
video bitstrcarn con be included in the user data video packet. However, this 
disadvantageousJy results in a relatively large amount of overhead. One embodiment of tbe 
invention advactaseousry encodes FEC codes from only a selected portion of the data in the 
video bttstream. The user data header code in the user data video packet can further 
identify the selected data to which the ccjrresporidmg FEC codes apply. In one cxomple, 
FEC codes are provided and decoded only for data corresponding to at last one of motion 
vectors, DC coefficients, and header information. 

Figure 19 is a flowchart 1900 generally iUustxatmg a process of decoding 
systematically encoded FEC data in a video bttstream. The process can be activated onco 
per VOP. The decoding process is advantageously compatible with video bitstreams that 
include FEC coding andfliose that do not The process stmts at a first state 1904, where me 
process receives tho video bitstrcarn. The video bitstream can bo received wirebssly. 
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through a local or a remote network, and can farther be temporarily stored in bufibs and 
tbc like. The process advances from the find, state 1904 to a second state 1908. 

lo the second state 1908, the process retrieves tbc data from the video bilstrcam. 
For example, in an MPEC-4 decoder, the process can identify those portions corresponding 
to standard MPEG-4 video data and those portions corresponnms to EEC codes. In one 
embodiment, the process retrieves tho FEC codes from a user data video packet The 
process advances from the second stale 1908 to a decision block 1912. 

In the decision block 1912, tbc process determines whether FEC codes are available 
to be oscd wiih the other data retrieved m Che second state 190$. When FEC codes are 
available, the process proceeds from the decision block 1912 to a third state 1916. 
Otherwise, the process proceeds from the decision block 1912 to a fourth state 1920. m 
another embodiment, the decision block 1912 instead determines whether an error is 
present in the received video Ntstream. It win be understood that the corresponding portion 
of the video bitstrtaxn that is inspected tor errors can be stored in a befler. When an error is 
detected, Ojc process proceeds from the decision block 1912 to the third state 1916. When 
no error is detected, tho process proceeds from the decision block 1912 to the fourth state 



In the third state 1916, the process decodes the FEC codes to reconstruct the malty 
data and/or verify the correctness of the received data. The third state 1916 can include the 
decoding of the normal video data mat is accomparJcd with the FEC codes. In one 
embodiment, only selected portions of the video data supplemented with FEC codes, and 
the process reads header codes or the like, which indicate the data to which the retrieved 
FEC codes correspond. 

The process advances from the third state to on optional fifth state 1924. One 
encoding process farther includes other data m (he same packet as the EEC codes. For 
example, this other data can correspond to at least one of a count of the number of motion 
vectors, a count of the number of bits per packet that are encoded between the resync fieJd 
and the motion marker field. This count allows a decoder to advantageously res>nchronizc 
to a video bitstrcam earlier Hum at a place in a hiwrcam with the next marlccr met permits 
resynchrorrhation The process advances from the optional fifth sure 1924 to the end. The 
process can be reactivated lo process the next batch of data, such as another VOP. 



1920. 
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In Ihc fourth suae 1920, the process uses the retrieved video data. The retrieved 
data can be the normal video data corresponding to a video bitstrcam without embedded 
FKC codes. The retrieved data can also correspond the norma) video data thai is 
maintained icpnraiely in the video bitstrcam from the embedded FEC codes. The process 
then ends until reactivated to process the next batch of data. 

Figure 20 is a block diagram generally illustrating one process of using a ring buffer 
ia error resilient decoding of video data. Data can be transmitted and/or received in varying 
bit rates and in bursts. For example, network congestion can cause delays in the receipt of 
packets of data The dropping of data, particularly in wireless environments, can also 
occur. In addition, a relatively small amount of received data can bo stored in a buffer until 
it is ready to be processed by a decoder. 

One embodiment of 4: invention edvanugcously uses a ring buffer to store 
incoming video bitstrcams for error rcsfliem decoding. A ring buffer is a buffer with a 
fixed size It will be understood that the size of the ring buffer can be selected in a very 
broad range A ring buffer can be constructed from an addressable memory, such as a 
random access memory (RAM). Another name for a ring buffer is circular buffer. 

The storing of the video bitstream in foe ring buffer is advantageous in error 
resilient decoding, including error resilient decoding of video mtstrezms in a wireless 
MPEG-4 compliant receiver, such as a video-enabled cellular r^^*^ With error 
resilient acceding techniques, data from the video bitstrcam may be read from the video 
bitstrcam multiple times, in multiple locations, and in multiple directions. The ring buffer 
rxrmits the decoder to retrieve data from various portions of the video bitsneam in a 
reliable and efficient manner. In one test, use of the ring buffer sped access to bitstrcam 
data by a factor of two. 

In contrast to other buffer implementations, data is advantageously not flushed from 
a ring buffer. Data enters and exits the ring buffer in a first-in first-out (FIFO) manner. 
When a ring buffer is fuU, the addition of an additional element ovenvrites the Dm element 
or the oldest element in the ring buffer. 

The block diagram of Figure 20 illustrates one configuration of a ring buffer 2002. 
Data received from the video bitstream is loaded into the ring buffer 2002 as the data is 
received, hi one embodiment, the modules of the <lecoder that decode (he video bitstrcam 
do not access the video btetrearo directly, but rather, access the video bitstrcam data that is 
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stored in the ring buffer 2002. Also, the skiDed practitioner will appreciate thai the ring 
buffer 2002 cau reside dtber ahead of or behind a VOP decoder in Iho data flow. However, 
the placement of the ring buffer 2002 ahead of the VOP decoder saves memory far the ring 
buffer 2002, as the VOP is in compressed form ahead of the VOP decoder. 

The video bhstrearo data that is loaded into the ring buffer 2002 is repr e s ented in 
Figure 20 by a bitstream file 2004. Data logging information, including error logging 
information, such as error flags, U aim stored in the ring buffer 2002 as it is generated. The 
data logging information is represented in Figure 20 as a log file 2006. In one embodiment, 
a log interface between rL223 output end decoder input advantageously synchronizes or 
aligns the data logging informal km in the ring buffer 2002 with the video bitstream data. 

A first arrow 2010 cor re sp onds to a location (address) in the ring buffer 2002 in 
which data is stored. As data is added to the ring buffer 2002, the ring buffer 2002 
conceptually routes in (he clockwise direction as shown in Figure 20. A second arrow 
2012 indicates an illustrative position from which data is retrieved from the ring buffer 
2002. A third arrow 2014 can correspond to ao illustrative byte position in the packet that 
is being retrieved or accessed. Packet start codes 201 6 can be dispersed throughout the ring 
buffer 2002. 

When data is retrieved from the ring buffer 2002 for decoding of a VOP with video 
packets enabled, one embodiment of the decoder inspect! the cocrespcading error-dug of 
each pocket. When the packets arc found to be ccOTrpted, the decoder stops the packets 
until the decoder eocounters a clean or error-free packet. When die decoder encounters a 
packet, it stores the arjprcpriflte location inJbrmation in an index table, which allows the 
decoder to access the packet efficiently without repeating a Beck for the packet In another 
embodiment, the decoder uses the contents of the ring buffer 2002 to recover and use data 
from partially corrupted video packets as described earlier in connection with Figures 7-16. 

Table II illustrates a sample of contents of an index table, which allows relatively 
efficient access to packets stored in the ring buffer 2002. 



Index - Table Entry 




Initial 
Value 


Dcscriptrons 
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Index - Tabic Entry 


Valid 0 

i 


Valid flag. A value of 1 indicates that vol id data 
corresponding to (his entry infoowcon exists in the 
ring buffer. 


Past 


0 


Past flag, 0 indicates thai this index has a current or 
future index. 


PBS 


0 


Start position of the packet, which indicates a position 
in the ring buffer. 


ErrorTypc 


0 


Error type. 


Size 


0 


Packet Size. 



Tabic D 



Various embodiments of the invention have been described above. Although this 
in venn on has been described with reference to these specific embodiments, the descriptions 
are intended to be illustrative of the invention and are not intended to be banting. Various 
modifications and applications may occur to those skilled m the art without departing from 
the true spirit and scopo of die invention as defined in the appended cMms. 
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1 . A video decoder that conceals errors received in a video bitstream, the video 
decoder comprising; 

an error detection circuit adapted to detect errors in the video biXstream; 

a memory device configured to provide on indication of an error in a portion 
of a video bitstream corresponding to a portion in an image; 

a control circuit configured to be responsive to an indication of the error in a 
first portion of the image, where the control circuit is further configured to detect if 
a second portion above the first portion in the image and if a third portion below the 
first portion in the image arc error-free, where die control circuit is further 
configured to interpolate between co r responding data in the second portion of the 
.image and arrrespoading data in the third portion of the data to conceal the error. 

2. The video decoder as defined in Claim 1. wherein the control circuit is 
further configured to determine when the first portion is at an upper boundary of the image 
and to copy ccwefipondiog data from the third portion of the data to the first potion to 
conceal the error. 

3. The video decoder os defined in Claim 1, wherein the control circuit is 
further configured to conceal the error by setting pixels in the first portion to gray. 

4. A vidso decoder that adaptivcry conoeala errors received in a video 
bit itrenrn, the video decoder comprising: 

a memory module adapted to maintain error values for sclcctrd portions of 
an image; 

a plurality of error resilience modules that generate images fn response tn 

errors; 

a prediction module adapted to generate a plurality or predictions of error 
values corresponding to the plurality of error resilience modules; 

a control module adapted receive an indication of an error in the video 
bitstream and, in response, to select en error resilience module from the error 
resilience module based on a comparison of the predictions of error valuer 

5. The video deco der as defined in Claim 4, further comprising a lookup table 
thai stores redetermined error values corresponding to error resilience techniques, where 
the lookup tabic is coupled to the prediction module 
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6. A vidto decoder that conceals errors received in a video bttsticam, the video 
decode? comprising? 

a memory module adapted to maintain error variances for selected portions 
of as image; 

a plurality cf error resilience modules that generate images in response to 

errors; 

a prediction module adapted to gen erate a plurality of weights corresponding 
to the plurality of error resilience modules; 

a control modulo adapted receive an indication of an error in the video 
bitstrtam and, in response, to combine outputs of selected error resilience modnles 
with the weights from the prediction module to conceal the error. 

7. The video decoder as defined in Claim 6, wherein the prediction module 
provides a weight dial is related to an inverse of an expected error. 

8. The video decoder as defined in Claim 6, whereto the prediction module 
provides n weight that provides o mtmmnl mean squared error. 

9. ' An optimizer circuit that selectively applies an error concealment techniqee 
from among a plurality of error concealment techniques comprising; 

means for maintaining an estimated error relating to at least a portion of an 

image; 

means for using the estimated error to generate a plurality of projected error 
estimates corresponding to application of an error concealment technique; and 

means for selecting the error concealment technique that provides the lowest 
projected error estrmalc 

10. A method of concealing errors in a video decoder conipniing: 
detecting an error in a first portion of a video bitstream that is intra-coded; 
determining that a second portion of an image above (he first portion and a 

third portion of the image below (be first portion are not corrrqxed; and 

interpolating pixels in the first portion between a first horizontal row of 
pixels in the second partum and a second horizontal row of pixels in the third 
portion to conceal errors when the second portion and the third portion are not 
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11. The method as defined in Gun 10, wherein the interpolating comprises 
linear interpolating. 

12. The method as defined m Claim 10. further comprising: 

determining that tho first portion corresponds ro mi upper boundary of the 
image; and 

copying the second horizontal row of pixels from the third portion when the 
third portion is not corrupted. 

13. The method aa defined in Claim 10, further comprising: 
determining thai the second portion is corrupted; 
determining that the third portion is not corrupted; and 

copying the second horizontal row of pixels from the third portion when the 
second portion is corrupted and the third portion is not corroptod. 

14. The method as defined in Claim 10, further comprising; 
determining (hat tin second portion is corrupted; 
determining that the third portion is corrupted; end 

setting pixels in the first portion to gray when the second portion and the 
third portion are corrupted. 

15. The method as defined in Claim 10, wherein the detected error relates to an 
error in a frame. 

16. The method as defined in Claim 1 0, wherein the detected error relates to an 
error in a video object plane (VOP). 

17. Tic method as defined in Claim 10. wherein the detected error relate* to an 
error in a macroblock. 

11 A method of concealing errors in a video decoder comprising: 

detecting an error In a first portion of a video bitstream that is predictive- 
coded; 

providing a substitute motion vector when the error relates to a standard 
motion vector, 

csiDg a first reference portion of a previous frame with the substitute motion 
vector to reconstruct when the fiat reierence portion is available; and 
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□sing a second reference portion of a second fame thai is prior to the 
previous Game wrien the first reference portion of the previous firamo is not 
available. 

19. The method u defined in Claim 18, farther comprising: 

using the standard motion vector from the video bitetrcam with the second 
reference portion of the second frame when the motion vector is available and the 
first reference portion is not available; and 

compensating a magnitude of the standard motion vector to account for a 
temporal difference between me previous frame and the second frame. 

20. The method as defined in Claim 19, wherein the compensating (he 
magnitude comprises multiplying by two when the second frame is a previous-previous 
frame. 

21. The method m defined in Claim 18, further comprising: 

using the substitute motion vector with the second reference portion of the 
second frame when die standard motion vector is not available and the first 
r ef e r ence portion is not available; and 

compensating a magnitude of the aubstitute motion vector to account for a 
temporal difference between the previous frame end the second frame. 

22. The method as defined in Claim 18, wherein the substitute motion vector U 
a copy of an adjacent motion vector. 

23. Ihe method as defined in Churn 18. wherein the substitute motion vector is 
interpolated from adjacent motion vectors. 

24. The method k defined in Claim 18, further cccnpriiing: 

determining that another motion vector from the video bitstrcam is 
available; and 

using the other motion vector in lieu of the standard motion vector and the 
substitute motion vector to conceal (he error. 

25. A method of edeprrvely producing a video imago comprising: 
receiving video data for n frame; 

dcterrmiiing whether the video data la intra-coded wpredi«rve<c<Ied; 
when the video data is intra-coded: 
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determining whether Utc btio-codoO video data corresponds 
to on error; 

concealing (he error when ihe urtra-codod video data 
corresponds lo Its error; 

setting an error value that is associated with si least a portion 
or the video packet to & first predetermined value when iho intra- 
coded video data corresponds ta the error, 

resetting the error value when no error for the mtra-coded 
video data is detected; and 

using the tmra-coded video data when do error fix the intra- 
coded vjdoo data is detected; 
when the video dale is rntdktrvo*coded, determining whether the predictive- 
coded video data corresporjds to an error, 

when the predictive-coded video data corresponds to an error 

Ming the predictive-coded video data when no error for the 
predJcirvc*coded video data is detected and the associated error value 
is reset; 

projecting a fiisi estimated error corresponding to use of the 
predictive-coded video data when no error is detected for the 
pTctf ctivc-codcd video data and the associated error value is not 
reset; 

projecting a second estimated error corresponding to use of a 
first predictive-coded error concealment techm'qce wheo no error is 
detected for the r^tdktrve-coded video data and the associated error 
value is not reset; 

selecting between (he use of the predictive-coded video dato 
and the use of the first rnedktive-coded error concealment technique 
based on a comparison between the first projected estimated error 
and the second projected estimated error; and 

updating the error value according to which ofthc predierivr> 
coded video data and the first predictive-coded error concealment 
technique is selected; 
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and when (he prcdictive-codod video daia corresponds to an error: 

applying a second predictive-coded error concealment 

updating the error value according to the second predictive- 
coded error concealment technique. 

26. The method as deOned in Claim 25, wherein the first predictive-coded error 
coQotalmcn! technique and the second predictive-coded error concealment technique are 
the same. 

27. Tho method as defined in Claim 25, wherein the projecting a second 
estimated error further comprise* projecting a plurality of estimated errors corresponding to 
a plurality of error concealment techniques for predictive coding, and wherein the selecting 
between the use of tho predictive-coded video data end (he use of the predictivc-codcd error 
concealment technique further comprises selecting among the use of the predictive-coded 
video data and (he use of an error concealment technique from the plurality of error 
concealment techniques based on the c or responding estimated error projections. 

28. The method as defined hi Claim 25, wherein the applying tho second 
predictive-coded error concealment technique farther comprises: 

projecting a plurality of estimated errors corresponding to a plurality of error 
concealment techniques for predictive coding; 

using (he projected estimate errors to select among the plurality of error 
concealment techniques; 

applying the selected error concealment technique; and 

adjusting the error value according to the selected error concealment 
technique. 

29. The method as defined in Claim 25, wherein (he video data is a njacrobloci. 

30. The method as defined in Claim 25, wherein the video data is a video object 
plane (VOP). 

31. The method as defined in Qahn 25, wherein the video data is a frame. 

32. The method as defined in Claho 25, further comprising normalizing the error 
value to a range between 0 to 255. 
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33. The method as defined b Claim 25, further cum prising multiplying die error 
vaJoe with a leaky vibe thai has a value of Jess than 1 in responso to an advancement in a 
frame sequence. 

34. The method as defined in Claim 33. wherein the leaky value is about 053. 

35. The method as defined in Cfcrim 25, further comprising maintaining the error 
value in a memory array, wherein an error value in (he array is associated with at feast one 
pixel in the image. 

36. The method at defined in Claim 25, further comprising maintaining the error 
value in a memory array, wherein each pixel in the image is associated with an error value 
in the array. 

37. A melhod of producing a video imago comprising; 
receiving data for a video frame; 

determmiog whether the video frame is a precJctrvc-coded frame or is an 
intra-coded frame; 

performing the following when the video frame is the piedictive-coded 

frame: 

determining whether a group of video data from the video frame corresponds 
to an error; 

when there is no error in the group of video data: 

determining whether the group of video data is intra-coded or 
predictive-coded; 

itfro-deccding the group of video data when the group of 
video data is intra coded; 

resetting an error variance associated with at least a portion of 
the group of video data when the group of video ditz is intra coded; 

using a first weighted sum to reconstruct a portion of on 
image ccxrespanding to the group of video data when the video data 
is intra coded, where die first weighted sum combines results of at 
least a first and a second technique; end 

updating the error variance according to the first weighted 
sum used to reconstruct the portion of the image; 
and wIkq mere is an error in the group of video data: 
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concealing the error is the portion of the image corresponding 
to the group of video data; and 

updating the error variance according to the error 
cone calm en! 

32. The method as defined in Claim 37, wherein the group of video data 
comprises a macros lock. 

39- The method as defined in Claim 37, wherein the group of video data 
comprises a video object plans (VOP). 

40. The method as defined in Claim 37, wherein the group of video data 
comprises missing data. 

41. The method as defined m Claim 37, wherein the concealing the enor further 
comprises using a second weighted sum to conceal the portion of the imago corresponding 
to the group of video data, where the second weighted sum combines results of at least at 
leasr two error concealing techniques. 

42. The method as defined in Claim 37, wherein the first weighted torn weighs 
the results of the first and the second technique according to values that are related to 
inverses of expected errors of the first and the second techniques. 

'43. The method as defined in Claim 37, wherein the first technique comprises 
constructing (he portion of the image from a first reference portion of a previous frame and 
me second technique comprises ccfistrocting the portion of the image from a second 
reference portion of a frame that is prior to the previous frame. 

44. The method as defined in Claim 37, wherein the second weighted sum 
weighs the results of the third and (be fourth error concealing techniques rrcenjdrng to 
inverses of expected errors of the third and the fourth error concealing techniques, 
respcctrvcly. 

45. The rnethod as defined in Claim 37, when the video frame is the predictive- 
cod cd frame, further comprising: 

receiving a next group of video data; and 

continuing execution of the method until the groups of video data are 
processed. 

46. The method as defined in Claim 37, further comprising: 
performing (lie following when the video frame is (he intra-coded frame: 
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determining whether a group o f daia from the uurrespomls to sit error; 
when there is no error in the group of video data: 

intrn-decoding the group of video data; and 
resetting an error variance associated with at least a portion of 
the group of video data; 
unl when there is errur in Iho group of video data: 

concealing the error in the portion of the image corresponding 
to the group of video data; and 

setting the error variance to a predetennined value, 

47. The method as defined in Claim 46, when the video frame is (he iiiLTJ-codrd 
frame, further comprising; 

receiving a next gronp of video data; and 

continuing execution of portions of the method corresponding to groups of 
data in an irura-decoded frame until the groups of video daia are processed. 

48. A method of selecting an error concealment technique from among a 
plurality of error concealment techniques comprising: 

maintaining an estimated error relating to or least a portico of an image; 

using the estimated error to generate a plurality of projected error estimates 
corresponding to application of on error concealment technique; and 

selecting the error cnnrrjirmmt technique that provides the lowest projected 
error estimate. 

49. The method as defined in Claim 4S. wherein an error concealment technique 
from the plurality of error concealment technique- comprises setting all corresponding pixels 
to gray. 

50. Ths method as defined in Claim 48, wherein an error concealment technique 
from the plurality of error concealment technique comprises mterpcJatmg between a 
previous frame and a subsequent frame. 

51. A circuit adapted to recover useful data from a video picket that is at least 
partially corrupted, die circuit comprising: 

a decoding circuit configured to decode & video packet in a forward direction 
and in a backward direction, where the decoding circoit detects bit locations of 
errors first encmmiered in the fcoward dfrection and m 
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a couoier adapted 10 maintain a count of complete macrob tacts decoded in 
the forward dircctioc and in the backward direction: and 

a control circuit adapted to discard at least a portion of the video packet that 
corresponds to an overlapping region, where the control circuit is further configured 
to further discard additional data corresponding to a backtrxlting amount when 
there is no overlapping region, where (he control circuit is further configured to 
discard information in incomplete macroblocks, and where the control circuit is 
ednptcd to permit use of al Lsasl a portion of the rcmQining data. 

52. The circuit as defined in Claim 51, wherein (he backtracking amount is 90 
bits in each direction. 

53. The circuit as defined in Claim SI, wherein the control circuit is further 
cenftgercd to discard an intra-coded macroblock from a partiatry corrupted video packet 

54. The circuit as defined in Claim 51, further comprising a dug buffer, which 
stores a video bttstream and is coupled to the decoding circuit such that the decoding circuit 
can access a video packet in both the forward direction and in the backward direction. 

55. A circuit adapted to recover useful data from a video packet (hat is at least 
partially corrupted, the circuit comprising: 

a data parsing circuit adapted to determine whether a video packet is 
encoded with data partitioning enabled; 

an error checking circuit configured to determine whether an error exists 
ahead of a motion marker of (ho video packet; and 

a decoder coupled to the data parsing circuit and to the error checking 
circuit, where the decoder is adapted to decode at least a portico of the data in the 
corrupted video packet ahead of the motion marker when data parsing circuit 
indicates that the video packet is encoded with data partitioning enabled and when 
the error checking circuit mdiones thai (ho error does not exist ahead of the motion 
marker. 

56. The circuit as defined in Claim 55, wherein the error checkma. circuit further 
comprises: 

a prediction circuit that generates a predicted location for tfcc motion marker, 
adrenh that detects an actual location of the motion marker; and 
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a comparison circa ii that indicates that the aror exists Acad of (he motion 
marker when the actual location and the predicted location do not match. 
57. A circuit adapted to recover useful data from a video packet that is at least 
partially corrupted; the circuit comprising: 

means for receiving the video packet; 

means for ending without recovering data when corruption is detected in at 
least ono of a video packet header of the video packet, a DC portion of the video 
pocket, and a motion vector portion of the video packet: 

means for initiating decoding of the video packet in a forward direction; 

means for maintaining a first count of a number of macroblocks decoded 
without error in the forward direction; 

means for storing codewords decoded in the forward direction; 

means fox storing a first bit location when an error is fust detected in the 
forward direction; 

means for initiating decoding of the video packet in a reverse direction; 

means for mambmring a second count of a enmhrr of macroblocks decoded 
without error in the reverse direction; 

means for storing codewords decoded in the reverse direction; 

means tbr storing a second bit location when an error is first delected in the 
reverse direction; 

means for del enn bring if there is an overlapping region, where the 
overlapping region corresponds to a region identified in both the forward direction 
andmlhcrevcnsdirecdcnash^geneiror; 

means for discarding the data in the overlapping region and fox using the 
data in a remaining portion of the video packet if there is an overlapping region; and 

means for <Ht ca n1 m g the dam between a first backtracking amount ahead of 
the first error location in the forward direction and o second backtracking amoont 
behind the second error location in the first location, and far recovering the 
remaining portion of the video packet if mere is no overlapping region. 
58. A method of recovering useful data from a video packet that has been 
corrupted, the method connmsing: 

receiving the video packet; 
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ending without recovering date when comrpticn is detected in a video packet 
header of the video packet; 

ending without recovering data when corruption is detected in a DC portion 
of the video packet; 

ending without recovering data when corruption is detected h a motion 
vector portion of the video packet; 

initiating decoding of the video packet in a forward direction; 

maintaining a Grsr count of s number of rruicrohlocks decoded without error 
in the forward direction; 

storing codewords decoded in tbc forward direction; 

storing a first bit location when an error is first detected in the forward 



miUroing deciding of the video packet in o reverse direction; 

maintaining a second count of a number of rnacroblocJa decoded without 
error in the reverse direction; 

storing codewords decoded in the reverse direction; 

aloring a second bit location when an error is first delected hi the reverse 
direction; 

determining if there is an overlapping rcjrfon, where the overlapping region 
correspond* to a region identified in both the forward direction and in the reverse 



if there is an overlapping region, discarding the data in the overlapping 
region and using the data in a remaining portion of the video packet; and 

if there is no overlapping region, discard ins the data borween u first 
backtracking amount ahead oflhe first error location in the forward direction nod a 
second backtracking amount behind the second error location in the first location, 
sad recovering the remaining portion of the video packet, 

59. The method as defined in Claim 58, wherein tbc first error location and the 
second error location correspond to bit locations. 

60. The method as defined in Ctnfrn 58, wherein the first error location end the 
second error location correspond to macroblock boundaries. 
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61. The method 03 defined in Claim 5S, wherein iho first backtracJcing omouut 
and the second backtracking amount ore each to a next valid macroblock bocodarv. 

62. The method as defined in Claim 58, wherein the first backtracking amount 
and the second backtracking amount ere about 90 bits. 

63. The method as defined in Claim 58, further comprising discarding recovered 
data from a corrupted video packet thai corresponds to an intra-coded macroblock. 

64. The method as defined in Claim 58, farther comprising: 
dctennming whether AC prediction was disabled by the encoder; 

using a recovered infra-coded macroblock if the intra-coded macroblock is 
recovered from a portion of the video packet that is ahead of a DC marker in the 
forward direction, where the video packet was encoded with AC prediction 
disabled; and 

otherwise discarding recovered data corresponding to an intra-coded 
macroblock. 

65. The method as defined in Claim 58, further comprising using recovered data 
corresponding to a fast intra-coded macroblock only if no other intra-coded macfoblock 
exists to the immediate left of the first intra-coded maorobtock and no other intra-coded 
macroblock exists immediately above the first mrra-coded macroblock in the image. 

66. . The method as defined in Claim 58, further comprising concealin g errors 
with gray pixels for portions of the video packet that were not recoverable. 

67. A method for recovering data in a corrupted video packet amrprising: 
inspecting the video packet to determine whether foe video packet was 

encoded with daia partitioning enabled; 

determining whether an error exists ahead of a motion marker of the video 
packet; and 

decoding si least n portion of the data in the corrupted video packet ahead of 
the motion marker when the video packet was encoded with data partitioning 
enabled and when foe error does not exist ahead of the motion marker. 

68. The method as defined in Claim 67, wherein the determining whether the 
error exists ahead of the motion marker farther comprises: 

predicting a location for the motion marker; 
detecting the motion marker; 
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comparing on actual location of the motion mariner lo the predicted location 
of the motion marker, 

determining that the error exists behind the motion marker when the actual 
location and the predicted location match; end 

determining thai the error exists ahead of the motion marker when the actual 
location and the predicted location do not match. 

69. The method as defined m Claim 67, wherein ihe portion of the data decoded 
includes decoding of motion vectors, 

70. The method as defined ra Claim 67, wherein the portion of the data decoded 
includes decoding of not-coded macrobtock flags. 

71 . The method ox defined in Claim 67, wherein the portion of the data decoded 
includes decoding of luminance (DC) information. 

73. A video decoder adapted to reconstruci corrupted video data comprising: 

a receiver circuit adapted to receive a video bhslream; 

a buffer coupled to the receiver circuit, where the buffer is adapted to Store 
at least a portion of Ihe video bitstmam; 

a parsing circuit adapted to distinguish video data firm forward error 
cancclicn (EEC) codes; 

on error monitoring circuit configured to detect corruption in the video data; 

and 

an FEC decoder adapted to receive the video data and the FEC codes, where 
the FEC decoder is configured to remove die corruption in the video data to which 
the FEC codes apply. 

73. The video decoder as defined in Claim 72, wherein the FEC decoder 
decodes FEC codes that correspond to Bose-Cbandhuri-Hoccpenghem (BCH) codex 

74. The video decoder as defined in Claim 7K wherein the buffer is a ring 

buffer. 

75. The video decoder as defined in Claim 72, wherein the parsing citcuil is 
configured to retrieve the video data fiom a packet for a video object piano (VOP) and 
retrieving the FEC codes from a user data video packet associated wim the VOP. 

76. A video decoder that decodes a video bitstream that includes forward error 
correction (FEC) codes, the video decoder comprising; 
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means for receiving the video bitstream, winch includes both video dam and 
PEC codes; 

neons for retrieving video data from the video bitstrcam; 
means for determining if there is corruption in a portion or the video data 
retrieved; 

means for retrieving FEC codes from the video bitstrcam in response to c 
detection of corruption; and 

means for using the TEC codes to reconstruct the portion of the video data 
such that the portion of the video data is reuuvered without corruption. 

77. A process of decoding a video bitstrcam that includes forward error 
correction (FEC) codes, the process comprising: 

receiving the video bitstream, which includes both video data and FEC 

codes; 

retrieving video data from the video bitstrcam; 
dctcrrnming if there is corruption in a portion of the video data retrieved; 
retrieving FEC codes from the video bitstream in response to a detection of 
corruption; and 

• using the FEC codes to reconstruct the portion of the video data such that the 
portion of the video data is recovered without corruption. 

78. The process as defined in Claim 77, wherein the FEC codes correspond to 
BosoOiandhrm-Hocqnenghem (BCH) codes, 

79. The process as defined in Claim 77, further comprising: 
storing the video bitstrescB in a buffer, 

retrieving the video data from the buffer when retrieving video data from the 
video bitstream; and 

retrieving the FEC codes from the buffer when reprieving the FEC codes 
from the video bitstrcam. 

8a The process as defined in Claim 19 % wherein the buffer is a ring boiler. 

81. The process as defined in Claim 77, further comprising retrieving the video 
data from a racto for a video ob^ 
data video packet associated with (he VOP. 
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81 Tho process as defined in Claim 77, farther conmrlsing receiving a header 
code that specifies a subset of video data to which the FEC codes correspond, and applying 
(he FEC coda only to the aubsd ofvideo data. 

83. The process as defined in Claim 77, farther comprising concealing an error 
in a corresponding pixel with a grey color pixel when tho portion of the video data cannot 
be recovered without corruption. 

84. A process of decoding Q video biUtream that includes forward error 
correction (FEC) codes, the process comprising: 

receiving the video bitstream, which includes both video data and FEC 

codes: 

retrieving video data from the video bitstream; 

determining if FEC codes that correspond to the retrieved video data are 
available; 

retrieving FEC codes from the video bitstream when the FEC codes arc 
available; and 

using the FEC codes to decode the portion of the video data such that the 
portion of tho video data is recovered witboot corruption. 

85. The process as defined In Claim 84, wherein the FEC codes correspond to 
Bose-Cbaudhuri-Hocqtierighem (BCH) codes. 

85. The process as defined io Claim 84, farther comprising: 
storing the video bitstream m a buffer, 

nrtrievmg the video data from the buffer when retrieving video data from the 
video bitstream; and 

retrieving the FEC codes from the buffer when retrieving the FEC codes 
from the video bitstream. 

87. The process as defined m Claim 86, wherein the buffer is a ring buffer. 

88. The process as defined in Claim 84, further comprising retrieving the video 
data from a packet for a video object plane (VOP) and retrieving the FEC codes from a user 
data video packet associated with the VOP. 

89. The process as defined in Claim 84, further ccanprismg nxcivmg & header 
code that specifies a sibeet of rideo dam to which the FEC codes correspond, and replying 
the FEC codes only to the subset ofvideo data. 
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90. A data buffer circuit for a video decoder comprising: 
a receiver circuit adapted to receive a video bitslrcam; 
a ring buffer adapted to store the video bitrtrcnm; and 

an error resilience module adapted to retrieve data from the ring buffer. 

91. The data buffer circuit as defined in Clam 90, wherein the receiver circuit 
comprises a wireless receiver. 

92. The data b offer circuit as defined in Clam 90, farther comprising a log 
interface circuit adapted to store data logging information in the ring b offer such that the 
daia logging information is aligned whh corresponding data from the video bimream. 

93. The data buffer circuit as defined in Clam 90. further comprising a VOP 
decoder disposed in a data flow between the receiver circuit and the ring buffer such that 
the video bimream stored by the ring buffer is in a decoded form. 

94. A data buffer circuit for a video decoder comprising: 
means for receiving a video hitstream; 

means for inspecting tbe video bitstreani for error; 

means for storing the video hitstream in a ring buffer regardless of an error 
indication; 

means for storing data logging information corresponding to video bitstream 
data in the ring buffer in an aligned manner with the ccarespc*ding video bitstream 
data; and 

means for automatically retrieving both a portion of the video bitstream and 
a corresponding portion of the data logging information from the ring buffer in 
response to a request for data. 

95. A method of accessing information from a video bitstream comprising: 
receiving e video bitstream; 

inspecting (be video bitstream for error; 

storing the video bilstremn in a ring buffer regardless of an error indication; 

storing data logging information ccoTespcndins v ideo bitslrann data in the 
ting biaTeT m an aligned rnanncr 

aimnnaticaU)' retrieving both a portion of the video bitstream and a 
corresponding portion of the data togging infbrntation from the ring buffer in 
response to arequest for data. 
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96. The method m defined in Claim 95, further comprising wirelessly receiving 
the video buslream. 

97. The method as defined m Claim 95, farther comprising receiving tho video 
bitstream in an MPEG-4 ccmpliam decoder. 

93. The method as defined in Claim 95. further comprising «l«rodtng video 
object planes (VOPg) from the video bitstream. prior to storing the video bitstream in the 
ring buffer, and wherein the storing of the video bitstrearo comprises storing the decoded 
VOPb. 

99. Tho method as defined in Claim 95, further comprising decoding video 
object planes (VOPs) from the video bitstream after the video bitstream has been stored in 
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